2023-07-20T02:11:00.304 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T02:11:00.305 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291 --owner scheduled_pdonnell@teuthology --verbose --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} --name pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/orig.config.yaml 2023-07-20T02:11:00.366 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T02:11:00.465 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291 branch: wip-pdonnell-testing-20230719.201220 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} email: pdonnell@redhat.com first_in_suite: false job_id: '7344291' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230719.201220 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230719.201220 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 6483 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 sleep_before_teardown: 0 subset: 383/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230719.201220 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi151.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa distributed 1 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/test_o_trunc.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 79491cb69dd37a1eafb93b3471998174bdfa2fd9 timestamp: 2023-07-20_00:39:06 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-20T02:11:00.466 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa; will attempt to use it 2023-07-20T02:11:00.467 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks 2023-07-20T02:11:00.467 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-20T02:11:00.469 INFO:teuthology.task.internal:Checking packages... 2023-07-20T02:11:00.497 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1b0b2eb005a7c51d97a788b9900d511e1394b537' 2023-07-20T02:11:00.497 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-20T02:11:00.498 INFO:teuthology.packaging:ref: None 2023-07-20T02:11:00.498 INFO:teuthology.packaging:tag: None 2023-07-20T02:11:00.498 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230719.201220 2023-07-20T02:11:00.498 INFO:teuthology.packaging:sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:11:00.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230719.201220 2023-07-20T02:11:01.175 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5040.g1b0b2eb0 2023-07-20T02:11:01.175 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-20T02:11:01.184 INFO:teuthology.task.internal:no buildpackages task found 2023-07-20T02:11:01.184 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-20T02:11:01.203 INFO:teuthology.task.internal:Saving configuration 2023-07-20T02:11:01.243 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-20T02:11:01.250 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:11:01.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:49.586796', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T02:11:01.306 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:49.584227', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T02:11:01.329 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:49.585686', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T02:11:01.330 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-20T02:11:01.337 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-20T02:11:01.337 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-20T02:11:01.338 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-20T02:11:01.338 INFO:teuthology.run_tasks:Running task console_log... 2023-07-20T02:11:01.463 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f277fb67430>, signals=[15]) 2023-07-20T02:11:01.465 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-20T02:11:01.473 INFO:teuthology.task.internal:Opening connections... 2023-07-20T02:11:01.475 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2023-07-20T02:11:01.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:11:01.548 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2023-07-20T02:11:01.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:11:01.618 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2023-07-20T02:11:01.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:11:01.689 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-20T02:11:01.698 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2023-07-20T02:11:01.721 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2023-07-20T02:11:01.722 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2023-07-20T02:11:01.781 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T02:11:01.782 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T02:11:01.782 INFO:teuthology.orchestra.run.smithi017.stdout:ID="rhel" 2023-07-20T02:11:01.782 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="fedora" 2023-07-20T02:11:01.782 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="8.6" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el8" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T02:11:01.783 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:11:01.784 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T02:11:01.784 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T02:11:01.784 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T02:11:01.784 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T02:11:01.785 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2023-07-20T02:11:01.816 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2023-07-20T02:11:01.838 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2023-07-20T02:11:01.838 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2023-07-20T02:11:01.897 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T02:11:01.897 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T02:11:01.898 INFO:teuthology.orchestra.run.smithi146.stdout:ID="rhel" 2023-07-20T02:11:01.898 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="fedora" 2023-07-20T02:11:01.898 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8.6" 2023-07-20T02:11:01.898 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2023-07-20T02:11:01.898 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T02:11:01.898 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2023-07-20T02:11:01.898 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T02:11:01.899 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T02:11:01.900 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2023-07-20T02:11:01.924 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2023-07-20T02:11:01.944 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2023-07-20T02:11:01.945 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2023-07-20T02:11:02.005 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T02:11:02.005 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T02:11:02.005 INFO:teuthology.orchestra.run.smithi151.stdout:ID="rhel" 2023-07-20T02:11:02.005 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="fedora" 2023-07-20T02:11:02.005 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="8.6" 2023-07-20T02:11:02.005 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el8" 2023-07-20T02:11:02.006 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T02:11:02.006 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2023-07-20T02:11:02.006 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T02:11:02.006 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T02:11:02.006 INFO:teuthology.orchestra.run.smithi151.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T02:11:02.006 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T02:11:02.007 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:11:02.007 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T02:11:02.007 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T02:11:02.007 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T02:11:02.007 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T02:11:02.007 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2023-07-20T02:11:02.032 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-20T02:11:02.045 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-20T02:11:02.056 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-20T02:11:02.057 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2023-07-20T02:11:02.059 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2023-07-20T02:11:02.061 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2023-07-20T02:11:02.080 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-20T02:11:02.088 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-20T02:11:02.089 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2023-07-20T02:11:02.118 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2023-07-20T02:11:02.120 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2023-07-20T02:11:02.148 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-20T02:11:02.240 INFO:teuthology.run_tasks:Running task kernel... 2023-07-20T02:11:02.259 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-07-20T02:11:02.260 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-07-20T02:11:02.261 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-07-20T02:11:02.261 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T02:11:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:11:02.281 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2023-07-20T02:11:02.342 INFO:teuthology.orchestra.run.smithi017.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T02:11:02.342 INFO:teuthology.task.kernel:Running kernel on smithi017: 4.18.0-372.9.1.el8.x86_64 2023-07-20T02:11:02.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2023-07-20T02:11:03.572 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-07-20T02:11:03.573 INFO:teuthology.orchestra.run.smithi017.stdout:Unable to read consumer identity 2023-07-20T02:11:05.451 INFO:teuthology.orchestra.run.smithi017.stdout:Copr repo for python3-asyncssh owned by ceph 5.8 kB/s | 3.5 kB 00:00 2023-07-20T02:11:44.214 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 413 kB/s | 16 MB 00:38 2023-07-20T02:11:47.815 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 443 kB/s | 24 kB 00:00 2023-07-20T02:11:48.608 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-20T02:11:48.648 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-07-20T02:11:48.649 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-07-20T02:11:48.649 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-07-20T02:11:48.767 DEBUG:teuthology.orchestra.run.smithi017:> echo no | sudo yum reinstall kernel || true 2023-07-20T02:11:49.120 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-07-20T02:11:49.120 INFO:teuthology.orchestra.run.smithi017.stdout:Unable to read consumer identity 2023-07-20T02:11:49.413 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 02:11:47 AM UTC. 2023-07-20T02:11:49.463 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T02:11:49.498 INFO:teuthology.orchestra.run.smithi017.stderr:Error: No packages marked for reinstall. 2023-07-20T02:11:49.546 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum reinstall -y kernel || true 2023-07-20T02:11:49.893 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-07-20T02:11:49.894 INFO:teuthology.orchestra.run.smithi017.stdout:Unable to read consumer identity 2023-07-20T02:11:50.170 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:03 ago on Thu 20 Jul 2023 02:11:47 AM UTC. 2023-07-20T02:11:50.213 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T02:11:50.238 INFO:teuthology.orchestra.run.smithi017.stderr:Error: No packages marked for reinstall. 2023-07-20T02:11:50.291 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2023-07-20T02:11:50.373 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-20T02:11:50.373 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-20T02:11:50.373 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-20T02:11:50.374 DEBUG:teuthology.parallel:result is None 2023-07-20T02:11:50.374 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-20T02:11:50.383 INFO:teuthology.task.internal:Creating test directory... 2023-07-20T02:11:50.383 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T02:11:50.385 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T02:11:50.387 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T02:11:50.408 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-20T02:11:50.420 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-20T02:11:50.436 INFO:teuthology.task.internal:Creating archive directory... 2023-07-20T02:11:50.436 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T02:11:50.442 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T02:11:50.447 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T02:11:50.485 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-20T02:11:50.496 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-20T02:11:50.497 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T02:11:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:11:50.536 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T02:11:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:11:50.555 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T02:11:50.557 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T02:11:50.578 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T02:11:50.593 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:11:50.609 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:11:50.617 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:11:50.634 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:11:50.635 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:11:50.650 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:11:50.652 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-20T02:11:50.661 INFO:teuthology.task.internal:Configuring sudo... 2023-07-20T02:11:50.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T02:11:50.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T02:11:50.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T02:11:50.722 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-20T02:11:50.735 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-20T02:11:50.736 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T02:11:50.738 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T02:11:50.747 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T02:11:50.780 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:11:50.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:11:50.886 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:11:50.973 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:11:51.011 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:11:51.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T02:11:51.079 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:11:51.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:11:51.177 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:11:51.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:11:51.305 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:11:51.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T02:11:51.374 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:11:51.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:11:51.478 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:11:51.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:11:51.610 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:11:51.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T02:11:51.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2023-07-20T02:11:51.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2023-07-20T02:11:51.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2023-07-20T02:11:51.738 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:11:51.747 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:11:51.792 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:11:52.195 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-20T02:11:52.204 INFO:teuthology.task.internal:Starting timer... 2023-07-20T02:11:52.204 INFO:teuthology.run_tasks:Running task pcp... 2023-07-20T02:11:52.221 INFO:teuthology.run_tasks:Running task selinux... 2023-07-20T02:11:52.237 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-07-20T02:11:52.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2023-07-20T02:11:52.304 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2023-07-20T02:11:52.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2023-07-20T02:11:52.378 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2023-07-20T02:11:52.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2023-07-20T02:11:52.446 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2023-07-20T02:11:52.448 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-20T02:11:52.449 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2023-07-20T02:11:52.477 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2023-07-20T02:11:52.478 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2023-07-20T02:11:52.501 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2023-07-20T02:11:52.501 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2023-07-20T02:11:52.529 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2023-07-20T02:11:52.530 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi151.front.sepia.ceph.com': 'permissive'} 2023-07-20T02:11:52.530 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:11:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:11:52.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:11:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:11:52.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:11:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:11:52.614 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-20T02:11:52.615 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2023-07-20T02:11:52.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2023-07-20T02:11:52.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2023-07-20T02:11:52.695 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-20T02:11:52.710 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-20T02:11:52.716 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-20T02:11:52.886 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-20T02:11:52.907 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-20T02:11:52.909 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi146.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-20T02:28:24.757 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2023-07-20T02:28:24.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-07-20T02:28:24.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:28:24.830 DEBUG:teuthology.orchestra.run.smithi017:> true 2023-07-20T02:28:24.901 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-07-20T02:28:24.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-07-20T02:28:24.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:28:24.971 DEBUG:teuthology.orchestra.run.smithi146:> true 2023-07-20T02:28:25.039 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-07-20T02:28:25.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-07-20T02:28:25.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:28:25.111 DEBUG:teuthology.orchestra.run.smithi151:> true 2023-07-20T02:28:25.182 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-07-20T02:28:25.183 INFO:teuthology.run_tasks:Running task clock... 2023-07-20T02:28:25.196 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-20T02:28:25.197 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T02:28:25.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:28:25.200 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T02:28:25.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:28:25.203 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T02:28:25.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:28:25.232 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T02:28:25.236 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T02:28:25.249 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T02:28:25.251 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T02:28:25.256 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T02:28:25.272 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T02:28:25.324 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2023-07-20T02:28:25.327 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2023-07-20T02:28:25.336 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2023-07-20T02:28:25.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2023-07-20T02:28:25.339 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2023-07-20T02:28:25.351 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2023-07-20T02:28:25.352 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T02:28:25.354 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T02:28:25.367 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T02:28:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T02:28:25.368 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T02:28:25.381 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T02:28:25.437 INFO:teuthology.orchestra.run.smithi017.stderr:bash: ntpq: command not found 2023-07-20T02:28:25.441 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:28:25.441 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2023-07-20T02:28:25.441 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.441 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.441 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.442 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.445 INFO:teuthology.orchestra.run.smithi146.stderr:bash: ntpq: command not found 2023-07-20T02:28:25.449 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:28:25.449 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2023-07-20T02:28:25.449 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.450 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.450 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.450 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:bash: ntpq: command not found 2023-07-20T02:28:25.458 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:28:25.458 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2023-07-20T02:28:25.459 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.459 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.459 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.459 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:28:25.460 INFO:teuthology.run_tasks:Running task pexec... 2023-07-20T02:28:25.470 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-20T02:28:25.471 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi017.front.sepia.ceph.com 2023-07-20T02:28:25.472 DEBUG:teuthology.orchestra.run.smithi017:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T02:28:25.473 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi146.front.sepia.ceph.com 2023-07-20T02:28:25.473 DEBUG:teuthology.orchestra.run.smithi146:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T02:28:25.474 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi151.front.sepia.ceph.com 2023-07-20T02:28:25.474 DEBUG:teuthology.orchestra.run.smithi151:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T02:28:25.863 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:25.936 INFO:teuthology.orchestra.run.smithi151.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:25.936 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:26.739 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:40 ago on Thu 20 Jul 2023 02:26:46 AM UTC. 2023-07-20T02:28:26.769 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:00 ago on Thu 20 Jul 2023 02:20:26 AM UTC. 2023-07-20T02:28:26.793 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:08:08 ago on Thu 20 Jul 2023 02:20:18 AM UTC. 2023-07-20T02:28:28.523 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-07-20T02:28:28.524 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-07-20T02:28:28.524 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2023-07-20T02:28:28.525 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-07-20T02:28:28.525 INFO:teuthology.orchestra.run.smithi146.stdout:Resetting modules: 2023-07-20T02:28:28.525 INFO:teuthology.orchestra.run.smithi146.stdout: container-tools 2023-07-20T02:28:28.525 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:28:28.525 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-07-20T02:28:28.525 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-07-20T02:28:28.525 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:28:28.655 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-07-20T02:28:28.656 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-07-20T02:28:28.656 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2023-07-20T02:28:28.656 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-07-20T02:28:28.657 INFO:teuthology.orchestra.run.smithi151.stdout:Resetting modules: 2023-07-20T02:28:28.657 INFO:teuthology.orchestra.run.smithi151.stdout: container-tools 2023-07-20T02:28:28.657 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:28:28.657 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-07-20T02:28:28.657 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-07-20T02:28:28.657 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:28:28.711 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-07-20T02:28:28.712 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-07-20T02:28:28.713 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2023-07-20T02:28:28.713 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-07-20T02:28:28.713 INFO:teuthology.orchestra.run.smithi017.stdout:Resetting modules: 2023-07-20T02:28:28.713 INFO:teuthology.orchestra.run.smithi017.stdout: container-tools 2023-07-20T02:28:28.713 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:28:28.713 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-07-20T02:28:28.714 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-07-20T02:28:28.714 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:28:29.499 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:43 ago on Thu 20 Jul 2023 02:26:46 AM UTC. 2023-07-20T02:28:29.640 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:03 ago on Thu 20 Jul 2023 02:20:26 AM UTC. 2023-07-20T02:28:29.710 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:08:11 ago on Thu 20 Jul 2023 02:20:18 AM UTC. 2023-07-20T02:28:32.933 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-07-20T02:28:32.964 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-07-20T02:28:33.357 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-07-20T02:28:33.411 INFO:teuthology.orchestra.run.smithi151.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:33.436 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:33.861 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:34.369 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:07 ago on Thu 20 Jul 2023 02:20:26 AM UTC. 2023-07-20T02:28:34.430 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:48 ago on Thu 20 Jul 2023 02:26:46 AM UTC. 2023-07-20T02:28:34.787 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:08:16 ago on Thu 20 Jul 2023 02:20:18 AM UTC. 2023-07-20T02:28:36.118 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-07-20T02:28:36.184 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-07-20T02:28:36.212 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================================ 2023-07-20T02:28:36.213 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-07-20T02:28:36.213 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================================ 2023-07-20T02:28:36.213 INFO:teuthology.orchestra.run.smithi151.stdout:Installing group/module packages: 2023-07-20T02:28:36.213 INFO:teuthology.orchestra.run.smithi151.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T02:28:36.213 INFO:teuthology.orchestra.run.smithi151.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T02:28:36.213 INFO:teuthology.orchestra.run.smithi151.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T02:28:36.214 INFO:teuthology.orchestra.run.smithi151.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T02:28:36.214 INFO:teuthology.orchestra.run.smithi151.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T02:28:36.214 INFO:teuthology.orchestra.run.smithi151.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T02:28:36.215 INFO:teuthology.orchestra.run.smithi151.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T02:28:36.215 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-07-20T02:28:36.215 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T02:28:36.215 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T02:28:36.216 INFO:teuthology.orchestra.run.smithi151.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T02:28:36.216 INFO:teuthology.orchestra.run.smithi151.stdout:Installing module profiles: 2023-07-20T02:28:36.216 INFO:teuthology.orchestra.run.smithi151.stdout: container-tools/common 2023-07-20T02:28:36.216 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:28:36.217 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling module streams: 2023-07-20T02:28:36.217 INFO:teuthology.orchestra.run.smithi151.stdout: container-tools rhel8 2023-07-20T02:28:36.217 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:28:36.218 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-07-20T02:28:36.218 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================================ 2023-07-20T02:28:36.218 INFO:teuthology.orchestra.run.smithi151.stdout:Install 10 Packages 2023-07-20T02:28:36.219 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:28:36.220 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 18 M 2023-07-20T02:28:36.220 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 65 M 2023-07-20T02:28:36.220 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-07-20T02:28:36.276 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================================ 2023-07-20T02:28:36.276 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-07-20T02:28:36.276 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================================ 2023-07-20T02:28:36.277 INFO:teuthology.orchestra.run.smithi146.stdout:Installing group/module packages: 2023-07-20T02:28:36.277 INFO:teuthology.orchestra.run.smithi146.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T02:28:36.277 INFO:teuthology.orchestra.run.smithi146.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T02:28:36.277 INFO:teuthology.orchestra.run.smithi146.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T02:28:36.277 INFO:teuthology.orchestra.run.smithi146.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T02:28:36.278 INFO:teuthology.orchestra.run.smithi146.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T02:28:36.278 INFO:teuthology.orchestra.run.smithi146.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T02:28:36.278 INFO:teuthology.orchestra.run.smithi146.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T02:28:36.278 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-07-20T02:28:36.278 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T02:28:36.278 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T02:28:36.278 INFO:teuthology.orchestra.run.smithi146.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T02:28:36.279 INFO:teuthology.orchestra.run.smithi146.stdout:Installing module profiles: 2023-07-20T02:28:36.279 INFO:teuthology.orchestra.run.smithi146.stdout: container-tools/common 2023-07-20T02:28:36.279 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:28:36.279 INFO:teuthology.orchestra.run.smithi146.stdout:Enabling module streams: 2023-07-20T02:28:36.279 INFO:teuthology.orchestra.run.smithi146.stdout: container-tools rhel8 2023-07-20T02:28:36.279 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:28:36.279 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-07-20T02:28:36.280 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================================ 2023-07-20T02:28:36.280 INFO:teuthology.orchestra.run.smithi146.stdout:Install 10 Packages 2023-07-20T02:28:36.280 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:28:36.280 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 18 M 2023-07-20T02:28:36.281 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 65 M 2023-07-20T02:28:36.281 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-07-20T02:28:36.302 INFO:teuthology.orchestra.run.smithi151.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2023-07-20T02:28:36.327 INFO:teuthology.orchestra.run.smithi151.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 222 kB/s | 25 kB 00:00 2023-07-20T02:28:36.352 INFO:teuthology.orchestra.run.smithi151.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 298 kB/s | 41 kB 00:00 2023-07-20T02:28:36.378 INFO:teuthology.orchestra.run.smithi151.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 654 kB/s | 49 kB 00:00 2023-07-20T02:28:36.411 INFO:teuthology.orchestra.run.smithi151.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-20T02:28:36.415 INFO:teuthology.orchestra.run.smithi146.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 181 kB/s | 25 kB 00:00 2023-07-20T02:28:36.436 INFO:teuthology.orchestra.run.smithi151.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-20T02:28:36.441 INFO:teuthology.orchestra.run.smithi146.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 251 kB/s | 41 kB 00:00 2023-07-20T02:28:36.491 INFO:teuthology.orchestra.run.smithi146.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 441 kB/s | 94 kB 00:00 2023-07-20T02:28:36.516 INFO:teuthology.orchestra.run.smithi146.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 492 kB/s | 49 kB 00:00 2023-07-20T02:28:36.575 INFO:teuthology.orchestra.run.smithi146.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-07-20T02:28:36.586 INFO:teuthology.orchestra.run.smithi151.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.2 MB/s | 213 kB 00:00 2023-07-20T02:28:36.642 INFO:teuthology.orchestra.run.smithi146.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-07-20T02:28:36.681 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-07-20T02:28:36.712 INFO:teuthology.orchestra.run.smithi151.stdout:(8/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 25 MB/s | 7.0 MB 00:00 2023-07-20T02:28:36.775 INFO:teuthology.orchestra.run.smithi146.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 28 MB/s | 8.0 MB 00:00 2023-07-20T02:28:36.783 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================================ 2023-07-20T02:28:36.784 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-07-20T02:28:36.784 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================================ 2023-07-20T02:28:36.785 INFO:teuthology.orchestra.run.smithi017.stdout:Installing group/module packages: 2023-07-20T02:28:36.785 INFO:teuthology.orchestra.run.smithi017.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T02:28:36.785 INFO:teuthology.orchestra.run.smithi017.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T02:28:36.785 INFO:teuthology.orchestra.run.smithi017.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T02:28:36.786 INFO:teuthology.orchestra.run.smithi017.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T02:28:36.786 INFO:teuthology.orchestra.run.smithi017.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T02:28:36.786 INFO:teuthology.orchestra.run.smithi017.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T02:28:36.787 INFO:teuthology.orchestra.run.smithi017.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T02:28:36.787 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-07-20T02:28:36.787 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T02:28:36.787 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T02:28:36.787 INFO:teuthology.orchestra.run.smithi017.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T02:28:36.788 INFO:teuthology.orchestra.run.smithi017.stdout:Installing module profiles: 2023-07-20T02:28:36.788 INFO:teuthology.orchestra.run.smithi017.stdout: container-tools/common 2023-07-20T02:28:36.788 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:28:36.788 INFO:teuthology.orchestra.run.smithi017.stdout:Enabling module streams: 2023-07-20T02:28:36.789 INFO:teuthology.orchestra.run.smithi017.stdout: container-tools rhel8 2023-07-20T02:28:36.789 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:28:36.789 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-07-20T02:28:36.789 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================================ 2023-07-20T02:28:36.790 INFO:teuthology.orchestra.run.smithi017.stdout:Install 10 Packages 2023-07-20T02:28:36.790 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:28:36.792 INFO:teuthology.orchestra.run.smithi151.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 11 MB/s | 2.2 MB 00:00 2023-07-20T02:28:36.793 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 18 M 2023-07-20T02:28:36.793 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 65 M 2023-07-20T02:28:36.793 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-07-20T02:28:36.800 INFO:teuthology.orchestra.run.smithi146.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 944 kB/s | 213 kB 00:00 2023-07-20T02:28:36.887 INFO:teuthology.orchestra.run.smithi151.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 15 MB/s | 8.0 MB 00:00 2023-07-20T02:28:36.888 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:28:36.888 INFO:teuthology.orchestra.run.smithi151.stdout:Total 27 MB/s | 18 MB 00:00 2023-07-20T02:28:36.895 INFO:teuthology.orchestra.run.smithi017.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 374 kB/s | 41 kB 00:00 2023-07-20T02:28:36.920 INFO:teuthology.orchestra.run.smithi017.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 185 kB/s | 25 kB 00:00 2023-07-20T02:28:36.946 INFO:teuthology.orchestra.run.smithi017.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 588 kB/s | 94 kB 00:00 2023-07-20T02:28:36.964 INFO:teuthology.orchestra.run.smithi146.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 22 MB/s | 7.0 MB 00:00 2023-07-20T02:28:36.971 INFO:teuthology.orchestra.run.smithi017.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 654 kB/s | 49 kB 00:00 2023-07-20T02:28:37.013 INFO:teuthology.orchestra.run.smithi017.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 5.6 MB/s | 524 kB 00:00 2023-07-20T02:28:37.026 INFO:teuthology.orchestra.run.smithi146.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 8.8 MB/s | 2.2 MB 00:00 2023-07-20T02:28:37.026 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:28:37.027 INFO:teuthology.orchestra.run.smithi146.stdout:Total 24 MB/s | 18 MB 00:00 2023-07-20T02:28:37.046 INFO:teuthology.orchestra.run.smithi017.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.9 MB/s | 149 kB 00:00 2023-07-20T02:28:37.080 INFO:teuthology.orchestra.run.smithi017.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 3.1 MB/s | 213 kB 00:00 2023-07-20T02:28:37.088 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-07-20T02:28:37.107 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-07-20T02:28:37.108 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-07-20T02:28:37.228 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-07-20T02:28:37.230 INFO:teuthology.orchestra.run.smithi017.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 15 MB/s | 2.2 MB 00:00 2023-07-20T02:28:37.247 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-07-20T02:28:37.248 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-07-20T02:28:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-07-20T02:28:37.294 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-07-20T02:28:37.339 INFO:teuthology.orchestra.run.smithi017.stdout:(9/10): buildah-1.26.2-1.module+el8.6.0+16771+2 20 MB/s | 8.0 MB 00:00 2023-07-20T02:28:37.429 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-07-20T02:28:37.433 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-07-20T02:28:37.438 INFO:teuthology.orchestra.run.smithi017.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 18 MB/s | 7.0 MB 00:00 2023-07-20T02:28:37.439 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:28:37.439 INFO:teuthology.orchestra.run.smithi017.stdout:Total 28 MB/s | 18 MB 00:00 2023-07-20T02:28:37.644 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-07-20T02:28:37.664 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-07-20T02:28:37.665 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-07-20T02:28:37.815 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-07-20T02:28:37.859 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-07-20T02:28:37.863 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-07-20T02:28:37.998 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-07-20T02:28:38.035 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T02:28:38.221 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T02:28:38.246 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:28:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:28:38.430 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-07-20T02:28:38.439 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T02:28:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T02:28:38.651 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T02:28:38.673 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T02:28:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:28:38.843 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T02:28:39.024 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T02:28:39.026 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T02:28:39.242 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T02:28:39.273 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T02:28:39.702 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T02:28:39.777 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T02:28:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T02:28:40.547 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T02:28:40.692 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T02:28:40.926 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T02:28:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T02:28:41.136 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T02:28:41.246 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T02:28:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:28:41.445 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T02:28:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T02:28:41.570 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:28:41.571 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T02:28:41.571 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:28:41.571 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T02:28:41.571 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T02:28:41.571 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T02:28:41.571 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T02:28:41.571 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T02:28:41.572 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T02:28:41.572 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T02:28:41.656 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:28:41.726 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T02:28:41.910 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T02:28:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:28:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T02:28:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:28:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T02:28:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T02:28:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T02:28:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T02:28:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T02:28:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T02:28:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T02:28:42.119 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:28:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:28:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T02:28:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:28:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T02:28:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T02:28:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T02:28:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T02:28:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T02:28:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T02:28:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T02:28:42.748 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T02:28:42.748 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:16 ago on Thu 20 Jul 2023 02:20:26 AM UTC. 2023-07-20T02:28:43.194 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T02:28:43.194 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:56 ago on Thu 20 Jul 2023 02:26:46 AM UTC. 2023-07-20T02:28:43.590 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T02:28:43.590 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:08:25 ago on Thu 20 Jul 2023 02:20:18 AM UTC. 2023-07-20T02:28:48.142 INFO:teuthology.orchestra.run.smithi151.stdout:Installed products updated. 2023-07-20T02:28:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:28:48.344 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-07-20T02:28:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:28:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T02:28:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T02:28:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T02:28:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:28:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T02:28:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:28:48.346 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-07-20T02:28:48.410 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-07-20T02:28:48.465 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-07-20T02:28:48.498 DEBUG:teuthology.parallel:result is None 2023-07-20T02:28:48.626 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:28:48.626 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-07-20T02:28:48.626 INFO:teuthology.orchestra.run.smithi146.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.627 INFO:teuthology.orchestra.run.smithi146.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:28:48.627 INFO:teuthology.orchestra.run.smithi146.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.627 INFO:teuthology.orchestra.run.smithi146.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T02:28:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T02:28:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T02:28:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:28:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T02:28:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:28:48.630 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-07-20T02:28:48.710 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:28:48.711 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-07-20T02:28:48.711 INFO:teuthology.orchestra.run.smithi017.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.711 INFO:teuthology.orchestra.run.smithi017.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:28:48.711 INFO:teuthology.orchestra.run.smithi017.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.711 INFO:teuthology.orchestra.run.smithi017.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T02:28:48.711 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T02:28:48.711 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T02:28:48.712 INFO:teuthology.orchestra.run.smithi017.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.712 INFO:teuthology.orchestra.run.smithi017.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:28:48.712 INFO:teuthology.orchestra.run.smithi017.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:28:48.712 INFO:teuthology.orchestra.run.smithi017.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T02:28:48.712 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:28:48.712 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-07-20T02:28:48.827 DEBUG:teuthology.parallel:result is None 2023-07-20T02:28:48.872 DEBUG:teuthology.parallel:result is None 2023-07-20T02:28:48.872 INFO:teuthology.run_tasks:Running task install... 2023-07-20T02:28:48.882 DEBUG:teuthology.task.install:project ceph 2023-07-20T02:28:48.882 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'}} 2023-07-20T02:28:48.883 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'} 2023-07-20T02:28:48.883 INFO:teuthology.task.install:Using flavor: default 2023-07-20T02:28:48.890 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-20T02:28:48.890 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-20T02:28:48.891 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T02:28:48.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:28:48.894 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T02:28:48.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:28:48.895 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T02:28:48.896 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:28:49.500 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T02:28:49.500 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T02:28:49.504 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T02:28:49.505 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T02:28:49.508 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T02:28:49.508 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T02:28:49.971 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T02:28:49.972 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:28:49.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T02:28:49.977 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T02:28:49.977 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:28:49.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T02:28:49.979 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T02:28:49.979 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:28:49.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T02:28:50.005 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T02:28:50.005 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T02:28:50.006 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T02:28:50.006 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T02:28:50.008 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T02:28:50.008 DEBUG:teuthology.orchestra.run.smithi017:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T02:28:50.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T02:28:50.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T02:28:50.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T02:28:50.150 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T02:28:50.151 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T02:28:50.154 DEBUG:teuthology.orchestra.run.smithi017:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T02:28:50.216 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2023-07-20T02:28:50.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2023-07-20T02:28:50.218 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2023-07-20T02:28:50.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2023-07-20T02:28:50.221 INFO:teuthology.orchestra.run.smithi017.stdout:check_obsoletes = 1 2023-07-20T02:28:50.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum clean all 2023-07-20T02:28:50.633 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:50.675 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:50.680 INFO:teuthology.orchestra.run.smithi151.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:51.351 INFO:teuthology.orchestra.run.smithi146.stdout:44 files removed 2023-07-20T02:28:51.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T02:28:51.505 INFO:teuthology.orchestra.run.smithi017.stdout:44 files removed 2023-07-20T02:28:51.517 INFO:teuthology.orchestra.run.smithi151.stdout:44 files removed 2023-07-20T02:28:51.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T02:28:51.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T02:28:51.828 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:51.915 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:51.936 INFO:teuthology.orchestra.run.smithi151.stdout:Updating Subscription Management repositories. 2023-07-20T02:28:53.933 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 87 kB/s | 81 kB 00:00 2023-07-20T02:28:53.955 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 81 kB/s | 81 kB 00:01 2023-07-20T02:28:53.983 INFO:teuthology.orchestra.run.smithi017.stdout:ceph packages for x86_64 86 kB/s | 81 kB 00:00 2023-07-20T02:28:54.745 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 12 kB/s | 9.4 kB 00:00 2023-07-20T02:28:54.819 INFO:teuthology.orchestra.run.smithi017.stdout:ceph noarch packages 12 kB/s | 9.4 kB 00:00 2023-07-20T02:28:54.898 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 10 kB/s | 9.4 kB 00:00 2023-07-20T02:28:55.589 INFO:teuthology.orchestra.run.smithi017.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-07-20T02:28:55.607 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 1.8 kB/s | 1.5 kB 00:00 2023-07-20T02:28:55.667 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-07-20T02:28:56.068 INFO:teuthology.orchestra.run.smithi017.stdout:Copr repo for python3-asyncssh owned by ceph 7.6 kB/s | 3.5 kB 00:00 2023-07-20T02:28:56.070 INFO:teuthology.orchestra.run.smithi151.stdout:Copr repo for python3-asyncssh owned by ceph 7.8 kB/s | 3.5 kB 00:00 2023-07-20T02:28:56.152 INFO:teuthology.orchestra.run.smithi146.stdout:Copr repo for python3-asyncssh owned by ceph 7.4 kB/s | 3.5 kB 00:00 2023-07-20T02:28:58.794 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 16 MB 00:02 2023-07-20T02:28:58.798 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 5.9 MB/s | 16 MB 00:02 2023-07-20T02:29:02.276 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 391 kB/s | 24 kB 00:00 2023-07-20T02:29:02.386 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 398 kB/s | 24 kB 00:00 2023-07-20T02:29:03.452 INFO:teuthology.orchestra.run.smithi146.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 47 MB 00:01 2023-07-20T02:29:03.562 INFO:teuthology.orchestra.run.smithi017.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 53 MB 00:01 2023-07-20T02:29:11.804 INFO:teuthology.orchestra.run.smithi146.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 MB/s | 53 MB 00:01 2023-07-20T02:29:12.456 INFO:teuthology.orchestra.run.smithi017.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-07-20T02:29:22.952 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:29:22.953 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:29:22.967 INFO:teuthology.orchestra.run.smithi146.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T02:29:22.968 INFO:teuthology.orchestra.run.smithi146.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T02:29:22.969 INFO:teuthology.orchestra.run.smithi146.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T02:29:22.970 INFO:teuthology.orchestra.run.smithi146.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T02:29:22.972 INFO:teuthology.orchestra.run.smithi146.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T02:29:22.975 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T02:29:22.976 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T02:29:23.109 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-07-20T02:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout:=========================================================================================================================== 2023-07-20T02:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-07-20T02:29:23.121 INFO:teuthology.orchestra.run.smithi146.stdout:=========================================================================================================================== 2023-07-20T02:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-07-20T02:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T02:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T02:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T02:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T02:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T02:29:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T02:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T02:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T02:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T02:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T02:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T02:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T02:29:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T02:29:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T02:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T02:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T02:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T02:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T02:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T02:29:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T02:29:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T02:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T02:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2023-07-20T02:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T02:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T02:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T02:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-07-20T02:29:23.127 INFO:teuthology.orchestra.run.smithi146.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T02:29:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T02:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T02:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T02:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T02:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T02:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T02:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:29:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T02:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T02:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T02:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T02:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T02:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T02:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T02:29:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T02:29:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T02:29:23.132 INFO:teuthology.orchestra.run.smithi146.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T02:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T02:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T02:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T02:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T02:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T02:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T02:29:23.133 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T02:29:23.134 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T02:29:23.135 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T02:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T02:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T02:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T02:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T02:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T02:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T02:29:23.136 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T02:29:23.137 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T02:29:23.138 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T02:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T02:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T02:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T02:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T02:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T02:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T02:29:23.139 INFO:teuthology.orchestra.run.smithi146.stdout:Enabling module streams: 2023-07-20T02:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql 10 2023-07-20T02:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-07-20T02:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout:=========================================================================================================================== 2023-07-20T02:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout:Install 128 Packages 2023-07-20T02:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 3 Packages 2023-07-20T02:29:23.140 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 206 M 2023-07-20T02:29:23.141 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-07-20T02:29:23.814 INFO:teuthology.orchestra.run.smithi146.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 29 kB/s | 6.4 kB 00:00 2023-07-20T02:29:24.065 INFO:teuthology.orchestra.run.smithi017.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:29:24.066 INFO:teuthology.orchestra.run.smithi017.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:29:24.080 INFO:teuthology.orchestra.run.smithi017.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T02:29:24.082 INFO:teuthology.orchestra.run.smithi017.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T02:29:24.083 INFO:teuthology.orchestra.run.smithi017.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T02:29:24.084 INFO:teuthology.orchestra.run.smithi017.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T02:29:24.086 INFO:teuthology.orchestra.run.smithi017.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T02:29:24.089 INFO:teuthology.orchestra.run.smithi017.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T02:29:24.090 INFO:teuthology.orchestra.run.smithi017.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T02:29:24.173 INFO:teuthology.orchestra.run.smithi146.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 2.3 MB/s | 850 kB 00:00 2023-07-20T02:29:24.233 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-07-20T02:29:24.246 INFO:teuthology.orchestra.run.smithi017.stdout:=========================================================================================================================== 2023-07-20T02:29:24.246 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-07-20T02:29:24.247 INFO:teuthology.orchestra.run.smithi017.stdout:=========================================================================================================================== 2023-07-20T02:29:24.247 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-07-20T02:29:24.247 INFO:teuthology.orchestra.run.smithi017.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T02:29:24.247 INFO:teuthology.orchestra.run.smithi017.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T02:29:24.247 INFO:teuthology.orchestra.run.smithi017.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T02:29:24.247 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T02:29:24.247 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T02:29:24.248 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T02:29:24.248 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T02:29:24.248 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T02:29:24.248 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T02:29:24.248 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T02:29:24.248 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T02:29:24.249 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T02:29:24.249 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T02:29:24.249 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T02:29:24.249 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T02:29:24.249 INFO:teuthology.orchestra.run.smithi017.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T02:29:24.249 INFO:teuthology.orchestra.run.smithi017.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T02:29:24.249 INFO:teuthology.orchestra.run.smithi017.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T02:29:24.250 INFO:teuthology.orchestra.run.smithi017.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T02:29:24.251 INFO:teuthology.orchestra.run.smithi017.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T02:29:24.251 INFO:teuthology.orchestra.run.smithi017.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T02:29:24.251 INFO:teuthology.orchestra.run.smithi017.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T02:29:24.251 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T02:29:24.251 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T02:29:24.251 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T02:29:24.251 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T02:29:24.252 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T02:29:24.252 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T02:29:24.252 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T02:29:24.252 INFO:teuthology.orchestra.run.smithi017.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T02:29:24.252 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrading: 2023-07-20T02:29:24.252 INFO:teuthology.orchestra.run.smithi017.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T02:29:24.252 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T02:29:24.253 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T02:29:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T02:29:24.255 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:29:24.255 INFO:teuthology.orchestra.run.smithi017.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T02:29:24.255 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T02:29:24.255 INFO:teuthology.orchestra.run.smithi017.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T02:29:24.255 INFO:teuthology.orchestra.run.smithi017.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T02:29:24.255 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T02:29:24.255 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T02:29:24.256 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T02:29:24.256 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T02:29:24.256 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T02:29:24.256 INFO:teuthology.orchestra.run.smithi017.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T02:29:24.256 INFO:teuthology.orchestra.run.smithi017.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T02:29:24.256 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T02:29:24.256 INFO:teuthology.orchestra.run.smithi017.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T02:29:24.257 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T02:29:24.258 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T02:29:24.258 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T02:29:24.258 INFO:teuthology.orchestra.run.smithi017.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T02:29:24.258 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T02:29:24.258 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T02:29:24.258 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T02:29:24.258 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T02:29:24.259 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T02:29:24.260 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T02:29:24.260 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T02:29:24.260 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:29:24.260 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T02:29:24.260 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T02:29:24.261 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T02:29:24.262 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T02:29:24.263 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:29:24.263 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T02:29:24.263 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T02:29:24.263 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T02:29:24.263 INFO:teuthology.orchestra.run.smithi017.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T02:29:24.263 INFO:teuthology.orchestra.run.smithi017.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T02:29:24.263 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T02:29:24.264 INFO:teuthology.orchestra.run.smithi017.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T02:29:24.265 INFO:teuthology.orchestra.run.smithi017.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T02:29:24.265 INFO:teuthology.orchestra.run.smithi017.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T02:29:24.265 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T02:29:24.265 INFO:teuthology.orchestra.run.smithi017.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T02:29:24.265 INFO:teuthology.orchestra.run.smithi017.stdout:Enabling module streams: 2023-07-20T02:29:24.265 INFO:teuthology.orchestra.run.smithi017.stdout: postgresql 10 2023-07-20T02:29:24.265 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:29:24.266 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-07-20T02:29:24.266 INFO:teuthology.orchestra.run.smithi017.stdout:=========================================================================================================================== 2023-07-20T02:29:24.266 INFO:teuthology.orchestra.run.smithi017.stdout:Install 128 Packages 2023-07-20T02:29:24.266 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrade 3 Packages 2023-07-20T02:29:24.266 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:29:24.267 INFO:teuthology.orchestra.run.smithi146.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 2.2 MB/s | 167 kB 00:00 2023-07-20T02:29:24.267 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 206 M 2023-07-20T02:29:24.267 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-07-20T02:29:24.457 INFO:teuthology.orchestra.run.smithi146.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 11 MB/s | 2.3 MB 00:00 2023-07-20T02:29:24.673 INFO:teuthology.orchestra.run.smithi146.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 7.2 MB/s | 1.6 MB 00:00 2023-07-20T02:29:24.999 INFO:teuthology.orchestra.run.smithi146.stdout:(6/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 14 MB/s | 4.6 MB 00:00 2023-07-20T02:29:25.000 INFO:teuthology.orchestra.run.smithi017.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2023-07-20T02:29:25.199 INFO:teuthology.orchestra.run.smithi146.stdout:(7/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 3.3 MB/s | 5.4 MB 00:01 2023-07-20T02:29:26.000 INFO:teuthology.orchestra.run.smithi017.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 850 kB/s | 850 kB 00:00 2023-07-20T02:29:26.159 INFO:teuthology.orchestra.run.smithi017.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.0 MB/s | 167 kB 00:00 2023-07-20T02:29:26.511 INFO:teuthology.orchestra.run.smithi146.stdout:(8/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x86 12 MB/s | 18 MB 00:01 2023-07-20T02:29:26.583 INFO:teuthology.orchestra.run.smithi146.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 342 kB/s | 24 kB 00:00 2023-07-20T02:29:27.309 INFO:teuthology.orchestra.run.smithi017.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 2.0 MB/s | 2.3 MB 00:01 2023-07-20T02:29:27.609 INFO:teuthology.orchestra.run.smithi017.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.9 MB/s | 5.4 MB 00:02 2023-07-20T02:29:27.808 INFO:teuthology.orchestra.run.smithi146.stdout:(10/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8 5.3 MB/s | 22 MB 00:04 2023-07-20T02:29:27.909 INFO:teuthology.orchestra.run.smithi146.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 2.5 MB/s | 260 kB 00:00 2023-07-20T02:29:27.935 INFO:teuthology.orchestra.run.smithi017.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 2.5 MB/s | 1.6 MB 00:00 2023-07-20T02:29:27.992 INFO:teuthology.orchestra.run.smithi146.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 373 kB/s | 31 kB 00:00 2023-07-20T02:29:28.192 INFO:teuthology.orchestra.run.smithi146.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 3.8 MB/s | 771 kB 00:00 2023-07-20T02:29:28.301 INFO:teuthology.orchestra.run.smithi146.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.3 MB/s | 139 kB 00:00 2023-07-20T02:29:28.401 INFO:teuthology.orchestra.run.smithi146.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.4 MB/s | 139 kB 00:00 2023-07-20T02:29:28.535 INFO:teuthology.orchestra.run.smithi146.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 3.5 MB/s | 472 kB 00:00 2023-07-20T02:29:28.753 INFO:teuthology.orchestra.run.smithi146.stdout:(17/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 3.8 MB/s | 14 MB 00:03 2023-07-20T02:29:28.877 INFO:teuthology.orchestra.run.smithi146.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 362 kB/s | 45 kB 00:00 2023-07-20T02:29:29.019 INFO:teuthology.orchestra.run.smithi146.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 866 kB/s | 122 kB 00:00 2023-07-20T02:29:29.136 INFO:teuthology.orchestra.run.smithi146.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.6 MB/s | 197 kB 00:00 2023-07-20T02:29:29.185 INFO:teuthology.orchestra.run.smithi017.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 2.9 MB/s | 4.6 MB 00:01 2023-07-20T02:29:29.353 INFO:teuthology.orchestra.run.smithi146.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.7 MB/s | 370 kB 00:00 2023-07-20T02:29:29.545 INFO:teuthology.orchestra.run.smithi146.stdout:(22/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 6.7 MB/s | 6.8 MB 00:01 2023-07-20T02:29:29.570 INFO:teuthology.orchestra.run.smithi146.stdout:(23/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 1.6 MB/s | 363 kB 00:00 2023-07-20T02:29:29.646 INFO:teuthology.orchestra.run.smithi146.stdout:(24/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.0 MB/s | 107 kB 00:00 2023-07-20T02:29:29.671 INFO:teuthology.orchestra.run.smithi146.stdout:(25/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 936 kB/s | 94 kB 00:00 2023-07-20T02:29:29.796 INFO:teuthology.orchestra.run.smithi146.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.4 MB/s | 179 kB 00:00 2023-07-20T02:29:29.913 INFO:teuthology.orchestra.run.smithi146.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 198 kB/s | 23 kB 00:00 2023-07-20T02:29:30.038 INFO:teuthology.orchestra.run.smithi146.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.0 MB/s | 129 kB 00:00 2023-07-20T02:29:30.164 INFO:teuthology.orchestra.run.smithi146.stdout:(29/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 6.8 MB/s | 3.5 MB 00:00 2023-07-20T02:29:30.989 INFO:teuthology.orchestra.run.smithi146.stdout:(30/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 11 MB/s | 48 MB 00:04 2023-07-20T02:29:31.057 INFO:teuthology.orchestra.run.smithi146.stdout:(31/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 1.7 MB/s | 1.7 MB 00:01 2023-07-20T02:29:31.090 INFO:teuthology.orchestra.run.smithi146.stdout:(32/131): ceph-mgr-modules-core-18.0.0-5040.g1b 2.4 MB/s | 241 kB 00:00 2023-07-20T02:29:31.123 INFO:teuthology.orchestra.run.smithi146.stdout:(33/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 721 kB/s | 47 kB 00:00 2023-07-20T02:29:31.157 INFO:teuthology.orchestra.run.smithi146.stdout:(34/131): ceph-prometheus-alerts-18.0.0-5040.g1 218 kB/s | 15 kB 00:00 2023-07-20T02:29:31.357 INFO:teuthology.orchestra.run.smithi146.stdout:(35/131): ceph-mgr-diskprediction-local-18.0.0- 6.2 MB/s | 7.4 MB 00:01 2023-07-20T02:29:31.432 INFO:teuthology.orchestra.run.smithi146.stdout:(36/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 600 kB/s | 165 kB 00:00 2023-07-20T02:29:31.458 INFO:teuthology.orchestra.run.smithi146.stdout:(37/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 179 kB/s | 18 kB 00:00 2023-07-20T02:29:31.483 INFO:teuthology.orchestra.run.smithi146.stdout:(38/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 718 kB/s | 258 kB 00:00 2023-07-20T02:29:31.842 INFO:teuthology.orchestra.run.smithi146.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 153 kB/s | 59 kB 00:00 2023-07-20T02:29:31.900 INFO:teuthology.orchestra.run.smithi146.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 180 kB/s | 75 kB 00:00 2023-07-20T02:29:31.975 INFO:teuthology.orchestra.run.smithi146.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 564 kB/s | 306 kB 00:00 2023-07-20T02:29:32.001 INFO:teuthology.orchestra.run.smithi146.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 435 kB/s | 44 kB 00:00 2023-07-20T02:29:32.076 INFO:teuthology.orchestra.run.smithi146.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-07-20T02:29:32.127 INFO:teuthology.orchestra.run.smithi146.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 262 kB/s | 33 kB 00:00 2023-07-20T02:29:32.168 INFO:teuthology.orchestra.run.smithi146.stdout:(45/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.4 MB/s | 476 kB 00:00 2023-07-20T02:29:32.194 INFO:teuthology.orchestra.run.smithi146.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 139 kB/s | 16 kB 00:00 2023-07-20T02:29:32.277 INFO:teuthology.orchestra.run.smithi146.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2023-07-20T02:29:32.302 INFO:teuthology.orchestra.run.smithi146.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 985 kB/s | 173 kB 00:00 2023-07-20T02:29:32.327 INFO:teuthology.orchestra.run.smithi146.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 607 kB/s | 81 kB 00:00 2023-07-20T02:29:32.411 INFO:teuthology.orchestra.run.smithi146.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-07-20T02:29:32.437 INFO:teuthology.orchestra.run.smithi146.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 162 kB/s | 18 kB 00:00 2023-07-20T02:29:32.512 INFO:teuthology.orchestra.run.smithi146.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 486 kB/s | 49 kB 00:00 2023-07-20T02:29:32.537 INFO:teuthology.orchestra.run.smithi146.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 586 kB/s | 59 kB 00:00 2023-07-20T02:29:32.619 INFO:teuthology.orchestra.run.smithi146.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 4.0 MB/s | 1.2 MB 00:00 2023-07-20T02:29:32.655 INFO:teuthology.orchestra.run.smithi146.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 424 kB/s | 60 kB 00:00 2023-07-20T02:29:32.680 INFO:teuthology.orchestra.run.smithi146.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-07-20T02:29:32.685 INFO:teuthology.orchestra.run.smithi017.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 2.8 MB/s | 22 MB 00:07 2023-07-20T02:29:32.722 INFO:teuthology.orchestra.run.smithi146.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 152 kB/s | 16 kB 00:00 2023-07-20T02:29:32.755 INFO:teuthology.orchestra.run.smithi146.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 335 kB/s | 34 kB 00:00 2023-07-20T02:29:32.782 INFO:teuthology.orchestra.run.smithi146.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-07-20T02:29:32.823 INFO:teuthology.orchestra.run.smithi146.stdout:(60/131): python3-rsa-4.8-1.el8.noarch.rpm 611 kB/s | 61 kB 00:00 2023-07-20T02:29:32.861 INFO:teuthology.orchestra.run.smithi017.stdout:(9/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el8 3.7 MB/s | 14 MB 00:03 2023-07-20T02:29:32.881 INFO:teuthology.orchestra.run.smithi146.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 149 kB/s | 19 kB 00:00 2023-07-20T02:29:32.894 INFO:teuthology.orchestra.run.smithi017.stdout:(10/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el 117 kB/s | 24 kB 00:00 2023-07-20T02:29:32.898 INFO:teuthology.orchestra.run.smithi146.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 204 kB/s | 24 kB 00:00 2023-07-20T02:29:32.923 INFO:teuthology.orchestra.run.smithi146.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2023-07-20T02:29:32.982 INFO:teuthology.orchestra.run.smithi146.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2023-07-20T02:29:33.003 INFO:teuthology.orchestra.run.smithi017.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 2.3 MB/s | 260 kB 00:00 2023-07-20T02:29:33.007 INFO:teuthology.orchestra.run.smithi146.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.3 MB/s | 255 kB 00:00 2023-07-20T02:29:33.033 INFO:teuthology.orchestra.run.smithi146.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-07-20T02:29:33.070 INFO:teuthology.orchestra.run.smithi017.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 466 kB/s | 31 kB 00:00 2023-07-20T02:29:33.124 INFO:teuthology.orchestra.run.smithi146.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 431 kB/s | 61 kB 00:00 2023-07-20T02:29:33.150 INFO:teuthology.orchestra.run.smithi146.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 196 kB/s | 23 kB 00:00 2023-07-20T02:29:33.175 INFO:teuthology.orchestra.run.smithi146.stdout:(69/131): python3-webtest-2.0.33-1.el8.noarch.r 510 kB/s | 85 kB 00:00 2023-07-20T02:29:33.200 INFO:teuthology.orchestra.run.smithi146.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-07-20T02:29:33.336 INFO:teuthology.orchestra.run.smithi017.stdout:(13/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 3.3 MB/s | 18 MB 00:05 2023-07-20T02:29:33.370 INFO:teuthology.orchestra.run.smithi017.stdout:(14/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 2.5 MB/s | 771 kB 00:00 2023-07-20T02:29:33.376 INFO:teuthology.orchestra.run.smithi146.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 652 kB/s | 110 kB 00:00 2023-07-20T02:29:33.404 INFO:teuthology.orchestra.run.smithi017.stdout:(15/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 2.0 MB/s | 139 kB 00:00 2023-07-20T02:29:33.418 INFO:teuthology.orchestra.run.smithi146.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 5.9 MB/s | 1.7 MB 00:00 2023-07-20T02:29:33.454 INFO:teuthology.orchestra.run.smithi017.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.6 MB/s | 139 kB 00:00 2023-07-20T02:29:33.485 INFO:teuthology.orchestra.run.smithi146.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 499 kB/s | 54 kB 00:00 2023-07-20T02:29:33.587 INFO:teuthology.orchestra.run.smithi017.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 2.5 MB/s | 472 kB 00:00 2023-07-20T02:29:33.654 INFO:teuthology.orchestra.run.smithi017.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 672 kB/s | 45 kB 00:00 2023-07-20T02:29:33.721 INFO:teuthology.orchestra.run.smithi017.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.8 MB/s | 122 kB 00:00 2023-07-20T02:29:33.726 INFO:teuthology.orchestra.run.smithi146.stdout:(74/131): python3-scipy-1.0.0-21.module+el8.5.0 25 MB/s | 14 MB 00:00 2023-07-20T02:29:33.751 INFO:teuthology.orchestra.run.smithi146.stdout:(75/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 776 kB/s | 259 kB 00:00 2023-07-20T02:29:33.777 INFO:teuthology.orchestra.run.smithi146.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.1 MB/s | 323 kB 00:00 2023-07-20T02:29:33.780 INFO:teuthology.orchestra.run.smithi017.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 3.3 MB/s | 197 kB 00:00 2023-07-20T02:29:33.861 INFO:teuthology.orchestra.run.smithi146.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 6.2 MB/s | 688 kB 00:00 2023-07-20T02:29:33.894 INFO:teuthology.orchestra.run.smithi146.stdout:(78/131): python3-markupsafe-0.23-19.el8.x86_64 232 kB/s | 39 kB 00:00 2023-07-20T02:29:33.905 INFO:teuthology.orchestra.run.smithi017.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.9 MB/s | 370 kB 00:00 2023-07-20T02:29:33.928 INFO:teuthology.orchestra.run.smithi146.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.5 MB/s | 538 kB 00:00 2023-07-20T02:29:33.978 INFO:teuthology.orchestra.run.smithi146.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2023-07-20T02:29:34.039 INFO:teuthology.orchestra.run.smithi017.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 2.7 MB/s | 363 kB 00:00 2023-07-20T02:29:34.087 INFO:teuthology.orchestra.run.smithi146.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 4.4 MB/s | 713 kB 00:00 2023-07-20T02:29:34.112 INFO:teuthology.orchestra.run.smithi146.stdout:(82/131): oniguruma-6.8.2-2.el8.x86_64.rpm 860 kB/s | 187 kB 00:00 2023-07-20T02:29:34.114 INFO:teuthology.orchestra.run.smithi017.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.4 MB/s | 107 kB 00:00 2023-07-20T02:29:34.189 INFO:teuthology.orchestra.run.smithi017.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 94 kB 00:00 2023-07-20T02:29:34.237 INFO:teuthology.orchestra.run.smithi146.stdout:(83/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-07-20T02:29:34.263 INFO:teuthology.orchestra.run.smithi146.stdout:(84/131): python3-jsonpatch-1.21-2.el8.noarch.r 151 kB/s | 27 kB 00:00 2023-07-20T02:29:34.321 INFO:teuthology.orchestra.run.smithi146.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.5 MB/s | 320 kB 00:00 2023-07-20T02:29:34.422 INFO:teuthology.orchestra.run.smithi146.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 239 kB/s | 44 kB 00:00 2023-07-20T02:29:34.588 INFO:teuthology.orchestra.run.smithi146.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 11 MB/s | 3.7 MB 00:00 2023-07-20T02:29:34.614 INFO:teuthology.orchestra.run.smithi146.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.1 MB/s | 225 kB 00:00 2023-07-20T02:29:34.639 INFO:teuthology.orchestra.run.smithi146.stdout:(89/131): python3-werkzeug-0.12.2-4.el8.noarch. 1.4 MB/s | 457 kB 00:00 2023-07-20T02:29:34.714 INFO:teuthology.orchestra.run.smithi146.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 510 kB/s | 64 kB 00:00 2023-07-20T02:29:34.806 INFO:teuthology.orchestra.run.smithi146.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 120 kB/s | 20 kB 00:00 2023-07-20T02:29:34.832 INFO:teuthology.orchestra.run.smithi146.stdout:(92/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 581 kB/s | 126 kB 00:00 2023-07-20T02:29:34.882 INFO:teuthology.orchestra.run.smithi146.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 619 kB/s | 103 kB 00:00 2023-07-20T02:29:34.982 INFO:teuthology.orchestra.run.smithi146.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.9 MB/s | 709 kB 00:00 2023-07-20T02:29:35.023 INFO:teuthology.orchestra.run.smithi017.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 4.2 MB/s | 3.5 MB 00:00 2023-07-20T02:29:35.057 INFO:teuthology.orchestra.run.smithi146.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.1 MB/s | 198 kB 00:00 2023-07-20T02:29:35.123 INFO:teuthology.orchestra.run.smithi017.stdout:(26/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 4.1 MB/s | 6.8 MB 00:01 2023-07-20T02:29:35.141 INFO:teuthology.orchestra.run.smithi146.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 16 MB/s | 4.9 MB 00:00 2023-07-20T02:29:35.172 INFO:teuthology.orchestra.run.smithi017.stdout:(27/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.2 MB/s | 179 kB 00:00 2023-07-20T02:29:35.199 INFO:teuthology.orchestra.run.smithi017.stdout:(28/131): ceph-grafana-dashboards-18.0.0-5040.g 307 kB/s | 23 kB 00:00 2023-07-20T02:29:35.241 INFO:teuthology.orchestra.run.smithi017.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.8 MB/s | 129 kB 00:00 2023-07-20T02:29:35.287 INFO:teuthology.orchestra.run.smithi146.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-07-20T02:29:35.325 INFO:teuthology.orchestra.run.smithi146.stdout:(98/131): jq-1.6-3.el8.x86_64.rpm 757 kB/s | 202 kB 00:00 2023-07-20T02:29:35.350 INFO:teuthology.orchestra.run.smithi146.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.7 MB/s | 354 kB 00:00 2023-07-20T02:29:35.475 INFO:teuthology.orchestra.run.smithi146.stdout:(100/131): postgresql-server-10.21-2.module+el8 27 MB/s | 5.1 MB 00:00 2023-07-20T02:29:35.550 INFO:teuthology.orchestra.run.smithi146.stdout:(101/131): postgresql-contrib-10.21-2.module+el 3.5 MB/s | 808 kB 00:00 2023-07-20T02:29:35.558 INFO:teuthology.orchestra.run.smithi017.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 4.8 MB/s | 1.7 MB 00:00 2023-07-20T02:29:35.576 INFO:teuthology.orchestra.run.smithi146.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 468 kB/s | 47 kB 00:00 2023-07-20T02:29:35.618 INFO:teuthology.orchestra.run.smithi146.stdout:(103/131): postgresql-10.21-2.module+el8.6.0+15 5.6 MB/s | 1.5 MB 00:00 2023-07-20T02:29:35.666 INFO:teuthology.orchestra.run.smithi017.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 2.2 MB/s | 241 kB 00:00 2023-07-20T02:29:35.685 INFO:teuthology.orchestra.run.smithi146.stdout:(104/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2023-07-20T02:29:35.702 INFO:teuthology.orchestra.run.smithi146.stdout:(105/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.3 MB/s | 200 kB 00:00 2023-07-20T02:29:35.727 INFO:teuthology.orchestra.run.smithi146.stdout:(106/131): python3-jwt-1.6.1-2.el8.noarch.rpm 282 kB/s | 43 kB 00:00 2023-07-20T02:29:35.758 INFO:teuthology.orchestra.run.smithi017.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 517 kB/s | 47 kB 00:00 2023-07-20T02:29:35.802 INFO:teuthology.orchestra.run.smithi146.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 410 kB/s | 48 kB 00:00 2023-07-20T02:29:35.834 INFO:teuthology.orchestra.run.smithi017.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 194 kB/s | 15 kB 00:00 2023-07-20T02:29:35.852 INFO:teuthology.orchestra.run.smithi146.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 578 kB/s | 87 kB 00:00 2023-07-20T02:29:35.878 INFO:teuthology.orchestra.run.smithi146.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 283 kB/s | 43 kB 00:00 2023-07-20T02:29:35.903 INFO:teuthology.orchestra.run.smithi146.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 1.1 MB/s | 109 kB 00:00 2023-07-20T02:29:35.929 INFO:teuthology.orchestra.run.smithi146.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.9 MB/s | 229 kB 00:00 2023-07-20T02:29:35.942 INFO:teuthology.orchestra.run.smithi017.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.3 MB/s | 258 kB 00:00 2023-07-20T02:29:35.971 INFO:teuthology.orchestra.run.smithi146.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 279 kB/s | 19 kB 00:00 2023-07-20T02:29:36.034 INFO:teuthology.orchestra.run.smithi017.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.8 MB/s | 165 kB 00:00 2023-07-20T02:29:36.037 INFO:teuthology.orchestra.run.smithi146.stdout:(113/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.4 MB/s | 560 kB 00:00 2023-07-20T02:29:36.063 INFO:teuthology.orchestra.run.smithi146.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.1 MB/s | 155 kB 00:00 2023-07-20T02:29:36.096 INFO:teuthology.orchestra.run.smithi146.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 4.4 MB/s | 559 kB 00:00 2023-07-20T02:29:36.109 INFO:teuthology.orchestra.run.smithi017.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 240 kB/s | 18 kB 00:00 2023-07-20T02:29:36.130 INFO:teuthology.orchestra.run.smithi146.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 194 kB/s | 18 kB 00:00 2023-07-20T02:29:36.172 INFO:teuthology.orchestra.run.smithi146.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.4 MB/s | 605 kB 00:00 2023-07-20T02:29:36.272 INFO:teuthology.orchestra.run.smithi146.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-07-20T02:29:36.297 INFO:teuthology.orchestra.run.smithi146.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 578 kB/s | 72 kB 00:00 2023-07-20T02:29:36.373 INFO:teuthology.orchestra.run.smithi146.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 9.6 MB/s | 2.3 MB 00:00 2023-07-20T02:29:36.385 INFO:teuthology.orchestra.run.smithi017.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-07-20T02:29:36.398 INFO:teuthology.orchestra.run.smithi146.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 597 kB/s | 60 kB 00:00 2023-07-20T02:29:36.423 INFO:teuthology.orchestra.run.smithi146.stdout:(122/131): fuse-2.9.7-15.el8.x86_64.rpm 550 kB/s | 83 kB 00:00 2023-07-20T02:29:36.443 INFO:teuthology.orchestra.run.smithi017.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-07-20T02:29:36.507 INFO:teuthology.orchestra.run.smithi146.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 734 kB/s | 98 kB 00:00 2023-07-20T02:29:36.510 INFO:teuthology.orchestra.run.smithi017.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-07-20T02:29:36.540 INFO:teuthology.orchestra.run.smithi146.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 406 kB/s | 58 kB 00:00 2023-07-20T02:29:36.599 INFO:teuthology.orchestra.run.smithi146.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 220 kB/s | 39 kB 00:00 2023-07-20T02:29:36.652 INFO:teuthology.orchestra.run.smithi017.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 3.3 MB/s | 476 kB 00:00 2023-07-20T02:29:36.666 INFO:teuthology.orchestra.run.smithi146.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.0 MB/s | 644 kB 00:00 2023-07-20T02:29:36.740 INFO:teuthology.orchestra.run.smithi146.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 861 kB/s | 171 kB 00:00 2023-07-20T02:29:36.786 INFO:teuthology.orchestra.run.smithi017.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 327 kB/s | 44 kB 00:00 2023-07-20T02:29:36.808 INFO:teuthology.orchestra.run.smithi146.stdout:(128/131): NetworkManager-initscripts-updown-1. 659 kB/s | 138 kB 00:00 2023-07-20T02:29:36.945 INFO:teuthology.orchestra.run.smithi017.stdout:(42/131): ceph-mgr-diskprediction-local-18.0.0- 4.4 MB/s | 7.4 MB 00:01 2023-07-20T02:29:36.970 INFO:teuthology.orchestra.run.smithi017.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.0 MB/s | 185 kB 00:00 2023-07-20T02:29:37.017 INFO:teuthology.orchestra.run.smithi146.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.7 MB/s | 798 kB 00:00 2023-07-20T02:29:37.055 INFO:teuthology.orchestra.run.smithi017.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 195 kB/s | 16 kB 00:00 2023-07-20T02:29:37.146 INFO:teuthology.orchestra.run.smithi017.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.8 MB/s | 173 kB 00:00 2023-07-20T02:29:37.171 INFO:teuthology.orchestra.run.smithi017.stdout:(46/131): python3-cachetools-3.1.1-4.el8.noarch 146 kB/s | 33 kB 00:00 2023-07-20T02:29:37.246 INFO:teuthology.orchestra.run.smithi017.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.8 MB/s | 384 kB 00:00 2023-07-20T02:29:37.330 INFO:teuthology.orchestra.run.smithi017.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2023-07-20T02:29:37.363 INFO:teuthology.orchestra.run.smithi017.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 423 kB/s | 81 kB 00:00 2023-07-20T02:29:37.405 INFO:teuthology.orchestra.run.smithi017.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2023-07-20T02:29:37.417 INFO:teuthology.orchestra.run.smithi146.stdout:(130/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 4.9 MB/s | 3.7 MB 00:00 2023-07-20T02:29:37.490 INFO:teuthology.orchestra.run.smithi017.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 585 kB/s | 49 kB 00:00 2023-07-20T02:29:37.565 INFO:teuthology.orchestra.run.smithi017.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 785 kB/s | 59 kB 00:00 2023-07-20T02:29:37.640 INFO:teuthology.orchestra.run.smithi017.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 805 kB/s | 60 kB 00:00 2023-07-20T02:29:37.715 INFO:teuthology.orchestra.run.smithi017.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 3.3 MB/s | 1.2 MB 00:00 2023-07-20T02:29:37.740 INFO:teuthology.orchestra.run.smithi017.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-07-20T02:29:37.782 INFO:teuthology.orchestra.run.smithi017.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 232 kB/s | 16 kB 00:00 2023-07-20T02:29:37.799 INFO:teuthology.orchestra.run.smithi017.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 574 kB/s | 34 kB 00:00 2023-07-20T02:29:37.834 INFO:teuthology.orchestra.run.smithi146.stdout:(131/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 3.3 MB/s | 3.6 MB 00:01 2023-07-20T02:29:37.837 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:29:37.838 INFO:teuthology.orchestra.run.smithi146.stdout:Total 14 MB/s | 206 MB 00:14 2023-07-20T02:29:37.858 INFO:teuthology.orchestra.run.smithi017.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 2.6 MB/s | 196 kB 00:00 2023-07-20T02:29:37.875 INFO:teuthology.orchestra.run.smithi017.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 817 kB/s | 61 kB 00:00 2023-07-20T02:29:37.917 INFO:teuthology.orchestra.run.smithi017.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 319 kB/s | 19 kB 00:00 2023-07-20T02:29:37.934 INFO:teuthology.orchestra.run.smithi017.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 408 kB/s | 24 kB 00:00 2023-07-20T02:29:37.976 INFO:teuthology.orchestra.run.smithi017.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 499 kB/s | 29 kB 00:00 2023-07-20T02:29:37.993 INFO:teuthology.orchestra.run.smithi017.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 470 kB/s | 27 kB 00:00 2023-07-20T02:29:38.068 INFO:teuthology.orchestra.run.smithi017.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.7 MB/s | 255 kB 00:00 2023-07-20T02:29:38.093 INFO:teuthology.orchestra.run.smithi017.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-07-20T02:29:38.143 INFO:teuthology.orchestra.run.smithi017.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 816 kB/s | 61 kB 00:00 2023-07-20T02:29:38.169 INFO:teuthology.orchestra.run.smithi017.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-07-20T02:29:38.236 INFO:teuthology.orchestra.run.smithi017.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 250 kB/s | 23 kB 00:00 2023-07-20T02:29:38.261 INFO:teuthology.orchestra.run.smithi017.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T02:29:38.378 INFO:teuthology.orchestra.run.smithi017.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 8.3 MB/s | 1.7 MB 00:00 2023-07-20T02:29:38.412 INFO:teuthology.orchestra.run.smithi017.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-07-20T02:29:38.495 INFO:teuthology.orchestra.run.smithi017.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 569 kB/s | 43 kB 00:00 2023-07-20T02:29:38.554 INFO:teuthology.orchestra.run.smithi017.stdout:(73/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.1 MB/s | 200 kB 00:00 2023-07-20T02:29:38.638 INFO:teuthology.orchestra.run.smithi017.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2023-07-20T02:29:38.679 INFO:teuthology.orchestra.run.smithi017.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 384 kB/s | 48 kB 00:00 2023-07-20T02:29:38.692 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-07-20T02:29:38.738 INFO:teuthology.orchestra.run.smithi017.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 870 kB/s | 87 kB 00:00 2023-07-20T02:29:38.780 INFO:teuthology.orchestra.run.smithi017.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 426 kB/s | 43 kB 00:00 2023-07-20T02:29:38.811 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-07-20T02:29:38.812 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-07-20T02:29:38.855 INFO:teuthology.orchestra.run.smithi017.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 933 kB/s | 109 kB 00:00 2023-07-20T02:29:38.880 INFO:teuthology.orchestra.run.smithi017.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2023-07-20T02:29:38.997 INFO:teuthology.orchestra.run.smithi017.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 160 kB/s | 19 kB 00:00 2023-07-20T02:29:39.039 INFO:teuthology.orchestra.run.smithi017.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.0 MB/s | 560 kB 00:00 2023-07-20T02:29:39.081 INFO:teuthology.orchestra.run.smithi017.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.8 MB/s | 155 kB 00:00 2023-07-20T02:29:39.223 INFO:teuthology.orchestra.run.smithi017.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-07-20T02:29:39.257 INFO:teuthology.orchestra.run.smithi017.stdout:(84/131): python3-cryptography-3.2.1-5.el8.x86_ 2.5 MB/s | 559 kB 00:00 2023-07-20T02:29:39.349 INFO:teuthology.orchestra.run.smithi017.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2023-07-20T02:29:39.416 INFO:teuthology.orchestra.run.smithi017.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-07-20T02:29:39.500 INFO:teuthology.orchestra.run.smithi017.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 15 MB/s | 2.3 MB 00:00 2023-07-20T02:29:39.558 INFO:teuthology.orchestra.run.smithi017.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 509 kB/s | 72 kB 00:00 2023-07-20T02:29:39.634 INFO:teuthology.orchestra.run.smithi017.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 619 kB/s | 83 kB 00:00 2023-07-20T02:29:39.692 INFO:teuthology.orchestra.run.smithi017.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 449 kB/s | 60 kB 00:00 2023-07-20T02:29:39.718 INFO:teuthology.orchestra.run.smithi017.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.1 MB/s | 98 kB 00:00 2023-07-20T02:29:39.793 INFO:teuthology.orchestra.run.smithi017.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 575 kB/s | 58 kB 00:00 2023-07-20T02:29:39.818 INFO:teuthology.orchestra.run.smithi017.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 385 kB/s | 39 kB 00:00 2023-07-20T02:29:39.893 INFO:teuthology.orchestra.run.smithi017.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 6.3 MB/s | 644 kB 00:00 2023-07-20T02:29:39.942 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-07-20T02:29:39.946 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-07-20T02:29:39.960 INFO:teuthology.orchestra.run.smithi017.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.2 MB/s | 171 kB 00:00 2023-07-20T02:29:40.019 INFO:teuthology.orchestra.run.smithi017.stdout:(96/131): NetworkManager-initscripts-updown-1.3 1.1 MB/s | 138 kB 00:00 2023-07-20T02:29:40.119 INFO:teuthology.orchestra.run.smithi017.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.1 MB/s | 110 kB 00:00 2023-07-20T02:29:40.361 INFO:teuthology.orchestra.run.smithi017.stdout:(98/131): python3-scipy-1.0.0-21.module+el8.5.0 34 MB/s | 14 MB 00:00 2023-07-20T02:29:40.387 INFO:teuthology.orchestra.run.smithi017.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 203 kB/s | 54 kB 00:00 2023-07-20T02:29:40.528 INFO:teuthology.orchestra.run.smithi017.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-07-20T02:29:40.554 INFO:teuthology.orchestra.run.smithi017.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2023-07-20T02:29:40.637 INFO:teuthology.orchestra.run.smithi017.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 358 kB/s | 39 kB 00:00 2023-07-20T02:29:40.687 INFO:teuthology.orchestra.run.smithi017.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 5.0 MB/s | 688 kB 00:00 2023-07-20T02:29:40.763 INFO:teuthology.orchestra.run.smithi017.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 4.2 MB/s | 538 kB 00:00 2023-07-20T02:29:40.813 INFO:teuthology.orchestra.run.smithi017.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-07-20T02:29:40.939 INFO:teuthology.orchestra.run.smithi017.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-07-20T02:29:40.980 INFO:teuthology.orchestra.run.smithi017.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 4.2 MB/s | 713 kB 00:00 2023-07-20T02:29:41.147 INFO:teuthology.orchestra.run.smithi017.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 159 kB/s | 27 kB 00:00 2023-07-20T02:29:41.222 INFO:teuthology.orchestra.run.smithi017.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-07-20T02:29:41.281 INFO:teuthology.orchestra.run.smithi017.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.3 MB/s | 320 kB 00:00 2023-07-20T02:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 310 kB/s | 44 kB 00:00 2023-07-20T02:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 6.7 MB/s | 457 kB 00:00 2023-07-20T02:29:41.524 INFO:teuthology.orchestra.run.smithi017.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2023-07-20T02:29:41.565 INFO:teuthology.orchestra.run.smithi017.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.6 MB/s | 225 kB 00:00 2023-07-20T02:29:41.633 INFO:teuthology.orchestra.run.smithi017.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 588 kB/s | 64 kB 00:00 2023-07-20T02:29:41.675 INFO:teuthology.orchestra.run.smithi017.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.1 MB/s | 126 kB 00:00 2023-07-20T02:29:41.750 INFO:teuthology.orchestra.run.smithi017.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 171 kB/s | 20 kB 00:00 2023-07-20T02:29:41.875 INFO:teuthology.orchestra.run.smithi017.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 516 kB/s | 103 kB 00:00 2023-07-20T02:29:42.086 INFO:teuthology.orchestra.run.smithi017.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 2.1 MB/s | 709 kB 00:00 2023-07-20T02:29:42.268 INFO:teuthology.orchestra.run.smithi017.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 12 MB/s | 4.9 MB 00:00 2023-07-20T02:29:42.293 INFO:teuthology.orchestra.run.smithi017.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 956 kB/s | 198 kB 00:00 2023-07-20T02:29:42.377 INFO:teuthology.orchestra.run.smithi017.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 2.4 MB/s | 202 kB 00:00 2023-07-20T02:29:42.477 INFO:teuthology.orchestra.run.smithi017.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-07-20T02:29:42.510 INFO:teuthology.orchestra.run.smithi017.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.6 MB/s | 354 kB 00:00 2023-07-20T02:29:42.644 INFO:teuthology.orchestra.run.smithi017.stdout:(125/131): postgresql-contrib-10.21-2.module+el 5.9 MB/s | 808 kB 00:00 2023-07-20T02:29:42.761 INFO:teuthology.orchestra.run.smithi017.stdout:(126/131): postgresql-server-10.21-2.module+el8 18 MB/s | 5.1 MB 00:00 2023-07-20T02:29:42.820 INFO:teuthology.orchestra.run.smithi017.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 8.6 MB/s | 1.5 MB 00:00 2023-07-20T02:29:43.245 INFO:teuthology.orchestra.run.smithi017.stdout:(128/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 4.6 MB/s | 48 MB 00:10 2023-07-20T02:29:43.445 INFO:teuthology.orchestra.run.smithi017.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.9 MB/s | 798 kB 00:00 2023-07-20T02:29:43.461 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 337 kB/s | 16 MB 00:47 2023-07-20T02:29:43.970 INFO:teuthology.orchestra.run.smithi017.stdout:(130/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 3.0 MB/s | 3.7 MB 00:01 2023-07-20T02:29:44.029 INFO:teuthology.orchestra.run.smithi017.stdout:(131/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 3.0 MB/s | 3.6 MB 00:01 2023-07-20T02:29:44.032 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:29:44.032 INFO:teuthology.orchestra.run.smithi017.stdout:Total 10 MB/s | 206 MB 00:19 2023-07-20T02:29:44.880 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-07-20T02:29:45.011 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-07-20T02:29:45.011 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-07-20T02:29:45.597 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-07-20T02:29:45.825 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T02:29:45.992 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T02:29:46.233 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-20T02:29:46.304 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-07-20T02:29:46.308 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-07-20T02:29:46.436 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T02:29:46.577 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T02:29:46.885 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T02:29:46.910 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 373 kB/s | 24 kB 00:00 2023-07-20T02:29:47.271 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T02:29:47.543 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:29:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:29:48.231 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:29:48.327 INFO:teuthology.orchestra.run.smithi151.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 MB/s | 47 MB 00:01 2023-07-20T02:29:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:29:48.618 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:29:48.716 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:29:48.910 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:29:49.181 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T02:29:49.430 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T02:29:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T02:29:50.172 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T02:29:50.368 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:29:50.504 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:29:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:29:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:29:50.976 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T02:29:51.119 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T02:29:51.312 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-07-20T02:29:51.326 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T02:29:51.540 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T02:29:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T02:29:51.901 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:29:51.935 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T02:29:51.995 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:29:52.193 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T02:29:52.225 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:29:52.678 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:29:52.778 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T02:29:53.184 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:29:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T02:29:53.617 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:29:53.825 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:29:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T02:29:53.957 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:29:54.142 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:29:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-20T02:29:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:29:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-20T02:29:54.439 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:29:54.627 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T02:29:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-20T02:29:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T02:29:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-20T02:29:55.088 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T02:29:55.162 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-20T02:29:55.454 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T02:29:55.644 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T02:29:55.674 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T02:29:55.804 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T02:29:55.863 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T02:29:56.067 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:29:56.175 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:29:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:29:56.320 INFO:teuthology.orchestra.run.smithi151.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 53 MB 00:01 2023-07-20T02:29:56.473 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:29:56.713 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T02:29:56.746 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T02:29:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T02:29:57.076 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T02:29:57.099 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-20T02:29:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:29:57.394 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-20T02:29:57.418 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:29:57.583 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:29:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-20T02:29:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T02:29:58.237 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:29:58.442 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:29:59.059 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:29:59.225 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:29:59.695 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T02:29:59.972 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:30:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-20T02:30:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T02:30:00.328 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T02:30:00.452 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T02:30:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T02:30:00.722 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T02:30:00.814 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T02:30:00.899 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T02:30:01.038 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T02:30:01.045 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T02:30:01.249 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-20T02:30:01.282 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T02:30:01.418 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-20T02:30:01.500 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:30:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-20T02:30:01.829 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-20T02:30:02.117 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-20T02:30:02.357 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-20T02:30:02.785 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-20T02:30:03.095 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-20T02:30:03.352 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-20T02:30:03.381 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:30:03.606 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-20T02:30:03.880 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-20T02:30:04.225 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-20T02:30:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T02:30:04.501 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-20T02:30:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-20T02:30:04.841 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T02:30:04.956 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-20T02:30:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T02:30:05.146 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-20T02:30:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-20T02:30:05.376 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T02:30:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T02:30:05.698 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-20T02:30:05.795 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T02:30:05.980 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-20T02:30:05.998 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T02:30:06.157 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T02:30:06.254 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-20T02:30:06.341 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T02:30:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-20T02:30:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T02:30:06.724 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:30:06.757 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-20T02:30:06.783 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:30:06.943 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-20T02:30:07.027 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T02:30:07.125 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T02:30:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T02:30:07.255 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T02:30:07.408 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:30:07.458 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-20T02:30:07.486 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:30:07.552 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:30:07.553 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:30:07.568 INFO:teuthology.orchestra.run.smithi151.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T02:30:07.569 INFO:teuthology.orchestra.run.smithi151.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T02:30:07.570 INFO:teuthology.orchestra.run.smithi151.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T02:30:07.571 INFO:teuthology.orchestra.run.smithi151.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T02:30:07.573 INFO:teuthology.orchestra.run.smithi151.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T02:30:07.576 INFO:teuthology.orchestra.run.smithi151.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T02:30:07.577 INFO:teuthology.orchestra.run.smithi151.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T02:30:07.710 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-07-20T02:30:07.720 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-20T02:30:07.722 INFO:teuthology.orchestra.run.smithi151.stdout:=========================================================================================================================== 2023-07-20T02:30:07.722 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-07-20T02:30:07.722 INFO:teuthology.orchestra.run.smithi151.stdout:=========================================================================================================================== 2023-07-20T02:30:07.722 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T02:30:07.723 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T02:30:07.724 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T02:30:07.724 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T02:30:07.724 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T02:30:07.724 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T02:30:07.724 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T02:30:07.724 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T02:30:07.724 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T02:30:07.725 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T02:30:07.725 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T02:30:07.725 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T02:30:07.725 INFO:teuthology.orchestra.run.smithi151.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T02:30:07.725 INFO:teuthology.orchestra.run.smithi151.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T02:30:07.725 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T02:30:07.725 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T02:30:07.726 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T02:30:07.727 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2023-07-20T02:30:07.728 INFO:teuthology.orchestra.run.smithi151.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T02:30:07.728 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T02:30:07.728 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T02:30:07.728 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-07-20T02:30:07.728 INFO:teuthology.orchestra.run.smithi151.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T02:30:07.728 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T02:30:07.728 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T02:30:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T02:30:07.730 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T02:30:07.730 INFO:teuthology.orchestra.run.smithi151.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T02:30:07.730 INFO:teuthology.orchestra.run.smithi151.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T02:30:07.730 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:30:07.730 INFO:teuthology.orchestra.run.smithi151.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T02:30:07.730 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T02:30:07.730 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T02:30:07.731 INFO:teuthology.orchestra.run.smithi151.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T02:30:07.731 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T02:30:07.731 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T02:30:07.731 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T02:30:07.731 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T02:30:07.731 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T02:30:07.731 INFO:teuthology.orchestra.run.smithi151.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T02:30:07.732 INFO:teuthology.orchestra.run.smithi151.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T02:30:07.732 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T02:30:07.732 INFO:teuthology.orchestra.run.smithi151.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T02:30:07.732 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T02:30:07.732 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T02:30:07.732 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T02:30:07.732 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T02:30:07.733 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T02:30:07.734 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T02:30:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T02:30:07.736 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T02:30:07.736 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T02:30:07.736 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T02:30:07.736 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T02:30:07.736 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T02:30:07.736 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T02:30:07.736 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T02:30:07.737 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T02:30:07.738 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T02:30:07.738 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:30:07.738 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T02:30:07.738 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T02:30:07.738 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T02:30:07.738 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T02:30:07.738 INFO:teuthology.orchestra.run.smithi151.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T02:30:07.739 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T02:30:07.740 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T02:30:07.740 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T02:30:07.740 INFO:teuthology.orchestra.run.smithi151.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T02:30:07.740 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T02:30:07.740 INFO:teuthology.orchestra.run.smithi151.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T02:30:07.740 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling module streams: 2023-07-20T02:30:07.740 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql 10 2023-07-20T02:30:07.741 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:30:07.741 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-07-20T02:30:07.741 INFO:teuthology.orchestra.run.smithi151.stdout:=========================================================================================================================== 2023-07-20T02:30:07.741 INFO:teuthology.orchestra.run.smithi151.stdout:Install 128 Packages 2023-07-20T02:30:07.741 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 3 Packages 2023-07-20T02:30:07.741 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:30:07.742 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 206 M 2023-07-20T02:30:07.742 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-07-20T02:30:07.767 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T02:30:07.929 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T02:30:07.950 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T02:30:08.025 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T02:30:08.132 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T02:30:08.138 INFO:teuthology.orchestra.run.smithi151.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2023-07-20T02:30:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-20T02:30:08.319 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T02:30:08.494 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T02:30:08.519 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-20T02:30:08.647 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T02:30:08.767 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T02:30:08.877 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T02:30:08.955 INFO:teuthology.orchestra.run.smithi151.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.0 MB/s | 850 kB 00:00 2023-07-20T02:30:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T02:30:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T02:30:09.147 INFO:teuthology.orchestra.run.smithi151.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 872 kB/s | 167 kB 00:00 2023-07-20T02:30:09.315 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T02:30:09.522 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T02:30:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T02:30:09.631 INFO:teuthology.orchestra.run.smithi151.stdout:(4/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 3.0 MB/s | 5.4 MB 00:01 2023-07-20T02:30:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T02:30:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T02:30:09.903 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T02:30:09.989 INFO:teuthology.orchestra.run.smithi151.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 4.3 MB/s | 1.6 MB 00:00 2023-07-20T02:30:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T02:30:10.169 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T02:30:10.311 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T02:30:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T02:30:10.473 INFO:teuthology.orchestra.run.smithi151.stdout:(6/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.7 MB/s | 2.3 MB 00:01 2023-07-20T02:30:10.574 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T02:30:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T02:30:10.757 INFO:teuthology.orchestra.run.smithi151.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 6.0 MB/s | 4.6 MB 00:00 2023-07-20T02:30:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T02:30:10.896 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T02:30:11.039 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T02:30:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T02:30:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T02:30:11.338 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T02:30:11.567 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T02:30:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T02:30:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T02:30:11.901 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T02:30:11.955 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T02:30:11.998 INFO:teuthology.orchestra.run.smithi151.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 5.4 MB/s | 22 MB 00:04 2023-07-20T02:30:12.132 INFO:teuthology.orchestra.run.smithi151.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 182 kB/s | 24 kB 00:00 2023-07-20T02:30:12.165 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T02:30:12.282 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T02:30:12.332 INFO:teuthology.orchestra.run.smithi151.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 8.6 MB/s | 14 MB 00:01 2023-07-20T02:30:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T02:30:12.474 INFO:teuthology.orchestra.run.smithi151.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.8 MB/s | 260 kB 00:00 2023-07-20T02:30:12.573 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T02:30:12.608 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T02:30:12.624 INFO:teuthology.orchestra.run.smithi151.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 207 kB/s | 31 kB 00:00 2023-07-20T02:30:12.843 INFO:teuthology.orchestra.run.smithi151.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 3.4 MB/s | 771 kB 00:00 2023-07-20T02:30:12.868 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T02:30:12.892 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T02:30:13.017 INFO:teuthology.orchestra.run.smithi151.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 803 kB/s | 139 kB 00:00 2023-07-20T02:30:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T02:30:13.183 INFO:teuthology.orchestra.run.smithi151.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 835 kB/s | 139 kB 00:00 2023-07-20T02:30:13.274 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T02:30:13.359 INFO:teuthology.orchestra.run.smithi151.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 2.6 MB/s | 472 kB 00:00 2023-07-20T02:30:13.496 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T02:30:14.042 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T02:30:14.064 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T02:30:14.126 INFO:teuthology.orchestra.run.smithi151.stdout:(17/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 8.8 MB/s | 6.8 MB 00:00 2023-07-20T02:30:14.276 INFO:teuthology.orchestra.run.smithi151.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 298 kB/s | 45 kB 00:00 2023-07-20T02:30:14.282 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T02:30:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T02:30:14.429 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T02:30:14.448 INFO:teuthology.orchestra.run.smithi151.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 715 kB/s | 122 kB 00:00 2023-07-20T02:30:14.586 INFO:teuthology.orchestra.run.smithi151.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.4 MB/s | 197 kB 00:00 2023-07-20T02:30:14.629 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T02:30:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T02:30:14.854 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T02:30:14.859 INFO:teuthology.orchestra.run.smithi151.stdout:(21/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 4.1 MB/s | 18 MB 00:04 2023-07-20T02:30:14.894 INFO:teuthology.orchestra.run.smithi151.stdout:(22/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.2 MB/s | 370 kB 00:00 2023-07-20T02:30:15.011 INFO:teuthology.orchestra.run.smithi151.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 913 kB/s | 107 kB 00:00 2023-07-20T02:30:15.036 INFO:teuthology.orchestra.run.smithi151.stdout:(24/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 2.0 MB/s | 363 kB 00:00 2023-07-20T02:30:15.051 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T02:30:15.055 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T02:30:15.153 INFO:teuthology.orchestra.run.smithi151.stdout:(25/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 661 kB/s | 94 kB 00:00 2023-07-20T02:30:15.295 INFO:teuthology.orchestra.run.smithi151.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.2 MB/s | 179 kB 00:00 2023-07-20T02:30:15.310 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T02:30:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T02:30:15.445 INFO:teuthology.orchestra.run.smithi151.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 154 kB/s | 23 kB 00:00 2023-07-20T02:30:15.494 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T02:30:15.579 INFO:teuthology.orchestra.run.smithi151.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 969 kB/s | 129 kB 00:00 2023-07-20T02:30:15.639 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T02:30:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T02:30:15.738 INFO:teuthology.orchestra.run.smithi151.stdout:(29/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 5.0 MB/s | 3.5 MB 00:00 2023-07-20T02:30:15.916 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T02:30:16.145 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T02:30:16.230 INFO:teuthology.orchestra.run.smithi151.stdout:(30/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 12 MB/s | 48 MB 00:04 2023-07-20T02:30:16.280 INFO:teuthology.orchestra.run.smithi151.stdout:(31/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 2.4 MB/s | 1.7 MB 00:00 2023-07-20T02:30:16.380 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T02:30:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:(32/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.6 MB/s | 241 kB 00:00 2023-07-20T02:30:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:(33/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 379 kB/s | 47 kB 00:00 2023-07-20T02:30:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T02:30:16.522 INFO:teuthology.orchestra.run.smithi151.stdout:(34/131): ceph-prometheus-alerts-18.0.0-5040.g1 102 kB/s | 15 kB 00:00 2023-07-20T02:30:16.539 INFO:teuthology.orchestra.run.smithi151.stdout:(35/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.9 MB/s | 258 kB 00:00 2023-07-20T02:30:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T02:30:16.665 INFO:teuthology.orchestra.run.smithi151.stdout:(36/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.1 MB/s | 165 kB 00:00 2023-07-20T02:30:16.690 INFO:teuthology.orchestra.run.smithi151.stdout:(37/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 119 kB/s | 18 kB 00:00 2023-07-20T02:30:16.757 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:30:16.832 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T02:30:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:30:17.033 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T02:30:17.090 INFO:teuthology.orchestra.run.smithi151.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 5.5 MB/s | 7.4 MB 00:01 2023-07-20T02:30:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T02:30:17.245 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T02:30:17.724 INFO:teuthology.orchestra.run.smithi151.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 57 kB/s | 59 kB 00:01 2023-07-20T02:30:18.099 INFO:teuthology.orchestra.run.smithi151.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 74 kB/s | 75 kB 00:01 2023-07-20T02:30:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 180 kB/s | 306 kB 00:01 2023-07-20T02:30:18.392 INFO:teuthology.orchestra.run.smithi151.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 150 kB/s | 44 kB 00:00 2023-07-20T02:30:18.493 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T02:30:18.567 INFO:teuthology.orchestra.run.smithi151.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 188 kB/s | 33 kB 00:00 2023-07-20T02:30:18.742 INFO:teuthology.orchestra.run.smithi151.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 93 kB/s | 16 kB 00:00 2023-07-20T02:30:18.809 INFO:teuthology.orchestra.run.smithi151.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 419 kB/s | 185 kB 00:00 2023-07-20T02:30:18.814 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T02:30:18.926 INFO:teuthology.orchestra.run.smithi151.stdout:(46/131): python3-asyncssh-2.7.0-2.el8.noarch.r 396 kB/s | 476 kB 00:01 2023-07-20T02:30:19.076 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T02:30:19.168 INFO:teuthology.orchestra.run.smithi151.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 405 kB/s | 173 kB 00:00 2023-07-20T02:30:19.194 INFO:teuthology.orchestra.run.smithi151.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 304 kB/s | 81 kB 00:00 2023-07-20T02:30:19.327 INFO:teuthology.orchestra.run.smithi151.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 70 kB/s | 11 kB 00:00 2023-07-20T02:30:19.352 INFO:teuthology.orchestra.run.smithi151.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 111 kB/s | 18 kB 00:00 2023-07-20T02:30:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T02:30:19.511 INFO:teuthology.orchestra.run.smithi151.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 308 kB/s | 49 kB 00:00 2023-07-20T02:30:19.670 INFO:teuthology.orchestra.run.smithi151.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 370 kB/s | 59 kB 00:00 2023-07-20T02:30:19.745 INFO:teuthology.orchestra.run.smithi151.stdout:(53/131): python3-cherrypy-18.4.0-1.el8.noarch. 411 kB/s | 384 kB 00:00 2023-07-20T02:30:19.751 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T02:30:19.845 INFO:teuthology.orchestra.run.smithi151.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 343 kB/s | 60 kB 00:00 2023-07-20T02:30:19.987 INFO:teuthology.orchestra.run.smithi151.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2023-07-20T02:30:20.008 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T02:30:20.121 INFO:teuthology.orchestra.run.smithi151.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 252 kB/s | 34 kB 00:00 2023-07-20T02:30:20.404 INFO:teuthology.orchestra.run.smithi151.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 429 kB/s | 283 kB 00:00 2023-07-20T02:30:20.538 INFO:teuthology.orchestra.run.smithi151.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 461 kB/s | 61 kB 00:00 2023-07-20T02:30:20.688 INFO:teuthology.orchestra.run.smithi151.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 125 kB/s | 19 kB 00:00 2023-07-20T02:30:20.814 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:30:20.830 INFO:teuthology.orchestra.run.smithi151.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 169 kB/s | 24 kB 00:00 2023-07-20T02:30:20.841 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T02:30:20.981 INFO:teuthology.orchestra.run.smithi151.stdout:(61/131): python3-routes-2.4.1-12.el8.noarch.rp 229 kB/s | 196 kB 00:00 2023-07-20T02:30:21.007 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:30:21.029 INFO:teuthology.orchestra.run.smithi151.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 148 kB/s | 29 kB 00:00 2023-07-20T02:30:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:30:21.126 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:30:21.139 INFO:teuthology.orchestra.run.smithi151.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 174 kB/s | 27 kB 00:00 2023-07-20T02:30:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T02:30:21.581 INFO:teuthology.orchestra.run.smithi151.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 462 kB/s | 255 kB 00:00 2023-07-20T02:30:21.679 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:30:21.690 INFO:teuthology.orchestra.run.smithi151.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 456 kB/s | 251 kB 00:00 2023-07-20T02:30:21.740 INFO:teuthology.orchestra.run.smithi151.stdout:(66/131): python3-kubernetes-11.0.0-6.el8.noarc 495 kB/s | 1.2 MB 00:02 2023-07-20T02:30:21.766 INFO:teuthology.orchestra.run.smithi151.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 333 kB/s | 61 kB 00:00 2023-07-20T02:30:21.882 INFO:teuthology.orchestra.run.smithi151.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 161 kB/s | 23 kB 00:00 2023-07-20T02:30:21.908 INFO:teuthology.orchestra.run.smithi151.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T02:30:21.983 INFO:teuthology.orchestra.run.smithi151.stdout:(70/131): python3-webtest-2.0.33-1.el8.noarch.r 291 kB/s | 85 kB 00:00 2023-07-20T02:30:22.038 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:30:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:30:22.125 INFO:teuthology.orchestra.run.smithi151.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 777 kB/s | 110 kB 00:00 2023-07-20T02:30:22.292 INFO:teuthology.orchestra.run.smithi151.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 325 kB/s | 54 kB 00:00 2023-07-20T02:30:22.323 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:30:22.484 INFO:teuthology.orchestra.run.smithi151.stdout:(73/131): python3-scipy-1.0.0-21.module+el8.5.0 24 MB/s | 14 MB 00:00 2023-07-20T02:30:22.511 INFO:teuthology.orchestra.run.smithi151.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-07-20T02:30:22.641 INFO:teuthology.orchestra.run.smithi151.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2023-07-20T02:30:22.670 INFO:teuthology.orchestra.run.smithi151.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 245 kB/s | 39 kB 00:00 2023-07-20T02:30:22.746 INFO:teuthology.orchestra.run.smithi151.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 6.4 MB/s | 688 kB 00:00 2023-07-20T02:30:22.791 INFO:teuthology.orchestra.run.smithi151.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 4.4 MB/s | 538 kB 00:00 2023-07-20T02:30:22.871 INFO:teuthology.orchestra.run.smithi151.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-07-20T02:30:22.905 INFO:teuthology.orchestra.run.smithi151.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-07-20T02:30:22.947 INFO:teuthology.orchestra.run.smithi151.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-07-20T02:30:23.122 INFO:teuthology.orchestra.run.smithi151.stdout:(82/131): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2023-07-20T02:30:23.148 INFO:teuthology.orchestra.run.smithi151.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 132 kB/s | 27 kB 00:00 2023-07-20T02:30:23.239 INFO:teuthology.orchestra.run.smithi151.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 480 kB/s | 44 kB 00:00 2023-07-20T02:30:23.290 INFO:teuthology.orchestra.run.smithi151.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.9 MB/s | 320 kB 00:00 2023-07-20T02:30:23.455 INFO:teuthology.orchestra.run.smithi151.stdout:(86/131): python3-numpy-1.14.3-10.el8.x86_64.rp 17 MB/s | 3.7 MB 00:00 2023-07-20T02:30:23.490 INFO:teuthology.orchestra.run.smithi151.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.2 MB/s | 457 kB 00:00 2023-07-20T02:30:23.515 INFO:teuthology.orchestra.run.smithi151.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.7 MB/s | 225 kB 00:00 2023-07-20T02:30:23.599 INFO:teuthology.orchestra.run.smithi151.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 590 kB/s | 64 kB 00:00 2023-07-20T02:30:23.858 INFO:teuthology.orchestra.run.smithi151.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 369 kB/s | 126 kB 00:00 2023-07-20T02:30:24.050 INFO:teuthology.orchestra.run.smithi151.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 44 kB/s | 20 kB 00:00 2023-07-20T02:30:24.075 INFO:teuthology.orchestra.run.smithi151.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 477 kB/s | 103 kB 00:00 2023-07-20T02:30:24.117 INFO:teuthology.orchestra.run.smithi151.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-07-20T02:30:24.234 INFO:teuthology.orchestra.run.smithi151.stdout:(94/131): openblas-threads-0.3.15-3.el8.x86_64. 31 MB/s | 4.9 MB 00:00 2023-07-20T02:30:24.309 INFO:teuthology.orchestra.run.smithi151.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.0 MB/s | 198 kB 00:00 2023-07-20T02:30:24.393 INFO:teuthology.orchestra.run.smithi151.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 2.4 MB/s | 202 kB 00:00 2023-07-20T02:30:24.485 INFO:teuthology.orchestra.run.smithi151.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2023-07-20T02:30:24.518 INFO:teuthology.orchestra.run.smithi151.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.8 MB/s | 354 kB 00:00 2023-07-20T02:30:24.635 INFO:teuthology.orchestra.run.smithi151.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 6.8 MB/s | 808 kB 00:00 2023-07-20T02:30:24.761 INFO:teuthology.orchestra.run.smithi151.stdout:(100/131): postgresql-server-10.21-2.module+el8 18 MB/s | 5.1 MB 00:00 2023-07-20T02:30:24.819 INFO:teuthology.orchestra.run.smithi151.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 8.2 MB/s | 1.5 MB 00:00 2023-07-20T02:30:24.836 INFO:teuthology.orchestra.run.smithi151.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 624 kB/s | 47 kB 00:00 2023-07-20T02:30:24.937 INFO:teuthology.orchestra.run.smithi151.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.7 MB/s | 200 kB 00:00 2023-07-20T02:30:24.962 INFO:teuthology.orchestra.run.smithi151.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 339 kB/s | 43 kB 00:00 2023-07-20T02:30:25.020 INFO:teuthology.orchestra.run.smithi151.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2023-07-20T02:30:25.046 INFO:teuthology.orchestra.run.smithi151.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 576 kB/s | 48 kB 00:00 2023-07-20T02:30:25.071 INFO:teuthology.orchestra.run.smithi151.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-07-20T02:30:25.129 INFO:teuthology.orchestra.run.smithi151.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 510 kB/s | 43 kB 00:00 2023-07-20T02:30:25.155 INFO:teuthology.orchestra.run.smithi151.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 1.3 MB/s | 109 kB 00:00 2023-07-20T02:30:25.189 INFO:teuthology.orchestra.run.smithi151.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2023-07-20T02:30:25.230 INFO:teuthology.orchestra.run.smithi151.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 7.3 MB/s | 560 kB 00:00 2023-07-20T02:30:25.233 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:30:25.256 INFO:teuthology.orchestra.run.smithi151.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 280 kB/s | 19 kB 00:00 2023-07-20T02:30:25.281 INFO:teuthology.orchestra.run.smithi151.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-07-20T02:30:25.339 INFO:teuthology.orchestra.run.smithi151.stdout:(114/131): libverto-devel-0.3.0-5.el8.x86_64.rp 307 kB/s | 18 kB 00:00 2023-07-20T02:30:25.373 INFO:teuthology.orchestra.run.smithi151.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 4.7 MB/s | 559 kB 00:00 2023-07-20T02:30:25.406 INFO:teuthology.orchestra.run.smithi151.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 8.9 MB/s | 605 kB 00:00 2023-07-20T02:30:25.432 INFO:teuthology.orchestra.run.smithi151.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-07-20T02:30:25.449 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:30:25.510 INFO:teuthology.orchestra.run.smithi151.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 928 kB/s | 72 kB 00:00 2023-07-20T02:30:25.566 INFO:teuthology.orchestra.run.smithi151.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 15 MB/s | 2.3 MB 00:00 2023-07-20T02:30:25.591 INFO:teuthology.orchestra.run.smithi151.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2023-07-20T02:30:25.616 INFO:teuthology.orchestra.run.smithi151.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-07-20T02:30:25.650 INFO:teuthology.orchestra.run.smithi151.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.6 MB/s | 98 kB 00:00 2023-07-20T02:30:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 861 kB/s | 58 kB 00:00 2023-07-20T02:30:25.725 INFO:teuthology.orchestra.run.smithi151.stdout:(124/131): thrift-0.13.0-2.el8.x86_64.rpm 449 kB/s | 1.7 MB 00:03 2023-07-20T02:30:25.751 INFO:teuthology.orchestra.run.smithi151.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 384 kB/s | 39 kB 00:00 2023-07-20T02:30:25.793 INFO:teuthology.orchestra.run.smithi151.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 5.8 MB/s | 644 kB 00:00 2023-07-20T02:30:25.818 INFO:teuthology.orchestra.run.smithi151.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.2 MB/s | 138 kB 00:00 2023-07-20T02:30:25.844 INFO:teuthology.orchestra.run.smithi151.stdout:(128/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.4 MB/s | 171 kB 00:00 2023-07-20T02:30:25.935 INFO:teuthology.orchestra.run.smithi151.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 8.5 MB/s | 798 kB 00:00 2023-07-20T02:30:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:30:26.379 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:30:26.462 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:30:26.632 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:30:26.721 INFO:teuthology.orchestra.run.smithi151.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 4.0 MB/s | 3.6 MB 00:00 2023-07-20T02:30:26.794 INFO:teuthology.orchestra.run.smithi151.stdout:(131/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 3.7 MB/s | 3.7 MB 00:01 2023-07-20T02:30:26.799 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:30:26.799 INFO:teuthology.orchestra.run.smithi151.stdout:Total 11 MB/s | 206 MB 00:19 2023-07-20T02:30:27.640 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-07-20T02:30:27.763 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-07-20T02:30:27.763 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-07-20T02:30:28.894 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-07-20T02:30:28.898 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-07-20T02:30:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-07-20T02:30:34.094 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T02:30:34.248 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T02:30:34.763 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-20T02:30:34.963 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T02:30:35.089 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T02:30:35.347 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T02:30:35.755 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T02:30:35.955 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:30:36.376 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:30:36.621 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:30:36.745 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:30:36.955 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:30:37.048 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:30:37.213 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:30:37.442 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T02:30:37.699 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T02:30:38.099 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T02:30:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T02:30:38.529 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:30:38.621 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:30:38.762 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:30:38.845 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:30:39.012 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T02:30:39.153 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T02:30:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T02:30:39.928 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T02:30:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T02:30:40.752 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T02:30:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:30:40.938 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2023-07-20T02:30:40.938 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2023-07-20T02:30:40.938 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /dev 2023-07-20T02:30:40.938 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /run 2023-07-20T02:30:40.938 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2023-07-20T02:30:40.939 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2023-07-20T02:30:40.939 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2023-07-20T02:30:40.939 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2023-07-20T02:30:40.939 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /dev 2023-07-20T02:30:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:30:41.004 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T02:30:41.338 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:30:41.716 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T02:30:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-20T02:30:42.239 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-20T02:30:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-20T02:30:42.660 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-20T02:30:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-20T02:30:42.926 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:30:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T02:30:43.152 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:30:43.302 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T02:30:43.372 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:30:43.466 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T02:30:43.596 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:30:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:30:44.221 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:30:44.238 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:30:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T02:30:44.711 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-20T02:30:44.991 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-20T02:30:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-20T02:30:45.454 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T02:30:45.973 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:30:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:30:46.126 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /sys 2023-07-20T02:30:46.126 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /proc 2023-07-20T02:30:46.126 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /dev 2023-07-20T02:30:46.126 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /run 2023-07-20T02:30:46.126 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /mnt 2023-07-20T02:30:46.126 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /var/tmp 2023-07-20T02:30:46.126 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /home 2023-07-20T02:30:46.127 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /tmp 2023-07-20T02:30:46.127 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /dev 2023-07-20T02:30:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:30:46.271 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:30:46.394 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T02:30:46.489 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:30:46.644 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T02:30:46.880 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:30:47.093 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:30:47.228 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T02:30:47.326 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:30:47.565 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-20T02:30:47.774 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T02:30:48.014 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T02:30:48.177 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:30:48.234 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T02:30:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:30:48.344 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:30:48.407 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T02:30:48.453 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:30:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:30:48.569 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-20T02:30:48.576 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:30:48.755 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:30:48.785 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-20T02:30:48.964 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T02:30:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-20T02:30:49.158 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-20T02:30:49.238 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:30:49.242 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:30:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:30:49.354 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-20T02:30:49.397 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:30:49.414 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:30:49.487 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:30:49.499 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:30:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-20T02:30:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-20T02:30:50.293 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-20T02:30:50.783 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-20T02:30:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T02:30:51.006 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-20T02:30:51.048 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:30:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:30:51.287 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:30:51.326 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-20T02:30:51.330 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:30:51.379 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:30:51.554 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:30:51.580 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:30:51.677 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-20T02:30:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T02:30:51.897 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T02:30:51.907 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:30:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-20T02:30:52.096 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:30:52.114 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-20T02:30:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:30:52.341 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-20T02:30:52.352 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:30:52.515 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-20T02:30:52.651 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:30:52.723 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-20T02:30:53.029 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-20T02:30:53.335 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-20T02:30:53.626 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-20T02:30:53.678 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:30:53.860 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-20T02:30:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:30:53.976 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:30:54.068 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-20T02:30:54.221 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-20T02:30:54.364 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T02:30:54.436 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T02:30:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T02:30:54.579 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:30:54.615 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:30:54.758 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-20T02:30:54.771 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:30:54.782 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:30:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-20T02:30:55.136 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T02:30:55.219 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T02:30:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-20T02:30:55.662 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-20T02:30:55.835 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T02:30:56.047 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T02:30:56.192 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T02:30:56.279 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T02:30:56.382 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:30:56.418 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:30:56.543 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T02:30:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:30:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:30:56.660 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:30:56.740 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T02:30:56.813 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:30:56.879 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T02:30:56.994 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T02:30:57.070 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T02:30:57.186 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T02:30:57.188 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:30:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T02:30:57.371 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:30:57.384 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T02:30:57.482 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T02:30:57.706 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T02:30:57.851 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T02:30:58.104 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T02:30:58.316 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T02:30:58.321 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-20T02:30:58.534 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T02:30:58.685 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-20T02:30:58.769 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T02:30:58.897 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 119/134 2023-07-20T02:30:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T02:30:59.148 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T02:30:59.195 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T02:30:59.382 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T02:30:59.438 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T02:30:59.587 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:31:00.581 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T02:31:00.907 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T02:31:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T02:31:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T02:31:01.819 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T02:31:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:02.022 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T02:31:02.219 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:02.236 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:02.241 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:31:02.520 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T02:31:02.777 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T02:31:02.847 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T02:31:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:31:03.096 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:31:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T02:31:03.207 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T02:31:03.311 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T02:31:03.554 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T02:31:03.640 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T02:31:03.752 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T02:31:03.761 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T02:31:03.924 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T02:31:03.975 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-20T02:31:04.194 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T02:31:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T02:31:04.280 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T02:31:04.345 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:31:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T02:31:04.549 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T02:31:04.550 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:31:04.797 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:04.798 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:31:04.808 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T02:31:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T02:31:05.068 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:31:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:31:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:05.810 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:05.811 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T02:31:05.811 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T02:31:05.811 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T02:31:05.811 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T02:31:05.811 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T02:31:05.812 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:31:05.812 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:31:05.812 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:31:05.812 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:31:05.812 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T02:31:05.812 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T02:31:05.813 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T02:31:05.813 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T02:31:05.813 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T02:31:05.813 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T02:31:05.813 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T02:31:05.813 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T02:31:05.814 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T02:31:05.814 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T02:31:05.814 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T02:31:05.814 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T02:31:05.814 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T02:31:05.814 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T02:31:05.814 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T02:31:05.815 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T02:31:05.815 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T02:31:05.815 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T02:31:05.815 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T02:31:05.815 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T02:31:05.815 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T02:31:05.822 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T02:31:05.822 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T02:31:05.822 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T02:31:05.823 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T02:31:05.824 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T02:31:05.824 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T02:31:05.824 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T02:31:05.824 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T02:31:05.824 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T02:31:05.824 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T02:31:05.824 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T02:31:05.825 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T02:31:05.825 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T02:31:05.825 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T02:31:05.825 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T02:31:05.825 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T02:31:05.825 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T02:31:05.826 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T02:31:05.826 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T02:31:05.826 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T02:31:05.826 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T02:31:05.826 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T02:31:05.826 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T02:31:05.826 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T02:31:05.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T02:31:05.828 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T02:31:05.828 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T02:31:05.828 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-20T02:31:05.828 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-20T02:31:05.828 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-20T02:31:05.828 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-20T02:31:05.828 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-20T02:31:05.829 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-20T02:31:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-20T02:31:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-20T02:31:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-20T02:31:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-20T02:31:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-20T02:31:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-20T02:31:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-20T02:31:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-20T02:31:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-20T02:31:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-20T02:31:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-20T02:31:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-20T02:31:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-20T02:31:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-20T02:31:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-20T02:31:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-20T02:31:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-20T02:31:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-20T02:31:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-20T02:31:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-20T02:31:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-20T02:31:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-20T02:31:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-20T02:31:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-20T02:31:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-20T02:31:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-20T02:31:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-20T02:31:05.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-20T02:31:05.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-20T02:31:05.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-20T02:31:05.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-20T02:31:05.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-20T02:31:05.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-20T02:31:05.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-20T02:31:05.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-20T02:31:05.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T02:31:05.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-20T02:31:05.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:31:05.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-20T02:31:05.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-20T02:31:05.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-20T02:31:05.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-20T02:31:05.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-20T02:31:05.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T02:31:05.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T02:31:05.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T02:31:05.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T02:31:05.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:05.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T02:31:07.059 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:31:07.319 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:31:07.560 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:07.829 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:07.847 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:07.916 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:31:08.030 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T02:31:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:31:08.305 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:31:08.321 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T02:31:08.414 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:08.414 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:57 ago on Thu 20 Jul 2023 02:29:11 AM UTC. 2023-07-20T02:31:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:31:08.629 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T02:31:08.988 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T02:31:09.153 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:31:09.240 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:31:09.470 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:31:09.484 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:31:09.637 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:31:09.646 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:31:09.789 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T02:31:09.789 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:10.024 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:10.024 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:31:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:31:10.445 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:10.994 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:10.995 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T02:31:10.995 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T02:31:10.995 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T02:31:10.995 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T02:31:10.996 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T02:31:10.996 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:31:10.996 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:31:10.996 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:31:10.996 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:31:10.996 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T02:31:10.997 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T02:31:10.997 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T02:31:10.997 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T02:31:10.997 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T02:31:10.997 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T02:31:10.997 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T02:31:10.997 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T02:31:10.998 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T02:31:10.999 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T02:31:10.999 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T02:31:10.999 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T02:31:10.999 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T02:31:10.999 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T02:31:10.999 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T02:31:11.007 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T02:31:11.007 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T02:31:11.008 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T02:31:11.008 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T02:31:11.009 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T02:31:11.010 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T02:31:11.010 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T02:31:11.010 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T02:31:11.011 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T02:31:11.011 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T02:31:11.011 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T02:31:11.011 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T02:31:11.011 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T02:31:11.011 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T02:31:11.012 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T02:31:11.012 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T02:31:11.012 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T02:31:11.012 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T02:31:11.012 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T02:31:11.012 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T02:31:11.013 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T02:31:11.013 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T02:31:11.013 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T02:31:11.013 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T02:31:11.013 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T02:31:11.013 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T02:31:11.013 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T02:31:11.014 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T02:31:11.014 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T02:31:11.014 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T02:31:11.014 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T02:31:11.014 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T02:31:11.015 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T02:31:11.015 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T02:31:11.015 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T02:31:11.015 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T02:31:11.016 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T02:31:11.016 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T02:31:11.016 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T02:31:11.016 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T02:31:11.017 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T02:31:11.017 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T02:31:11.017 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T02:31:11.017 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T02:31:11.018 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T02:31:11.018 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T02:31:11.018 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T02:31:11.018 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T02:31:11.018 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T02:31:11.018 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T02:31:11.019 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T02:31:11.019 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T02:31:11.019 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T02:31:11.019 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T02:31:11.019 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T02:31:11.020 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T02:31:11.020 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T02:31:11.020 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T02:31:11.020 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T02:31:11.020 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T02:31:11.021 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T02:31:11.021 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T02:31:11.021 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T02:31:11.021 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T02:31:11.021 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T02:31:11.021 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T02:31:11.022 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T02:31:11.022 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T02:31:11.022 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T02:31:11.022 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T02:31:11.022 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T02:31:11.022 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T02:31:11.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T02:31:11.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T02:31:11.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T02:31:11.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T02:31:11.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T02:31:11.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T02:31:11.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T02:31:11.024 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T02:31:11.024 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T02:31:11.024 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T02:31:11.024 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T02:31:11.024 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T02:31:11.024 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T02:31:11.024 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T02:31:11.025 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T02:31:11.025 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T02:31:11.025 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T02:31:11.025 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T02:31:11.025 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T02:31:11.025 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T02:31:11.025 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T02:31:11.026 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T02:31:11.026 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T02:31:11.026 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T02:31:11.026 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T02:31:11.026 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T02:31:11.026 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T02:31:11.027 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T02:31:11.027 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:11.027 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T02:31:13.463 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:13.464 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:02 ago on Thu 20 Jul 2023 02:29:11 AM UTC. 2023-07-20T02:31:16.872 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-07-20T02:31:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:31:17.086 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2023-07-20T02:31:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T02:31:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.086 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-07-20T02:31:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T02:31:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-3.el8.x86_64 2023-07-20T02:31:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T02:31:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T02:31:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T02:31:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T02:31:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T02:31:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T02:31:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T02:31:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T02:31:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T02:31:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T02:31:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T02:31:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T02:31:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T02:31:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T02:31:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T02:31:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T02:31:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T02:31:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T02:31:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T02:31:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T02:31:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T02:31:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T02:31:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T02:31:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T02:31:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T02:31:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T02:31:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T02:31:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T02:31:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T02:31:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T02:31:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T02:31:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T02:31:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T02:31:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T02:31:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T02:31:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T02:31:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-07-20T02:31:17.104 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-07-20T02:31:17.321 DEBUG:teuthology.parallel:result is None 2023-07-20T02:31:20.421 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-07-20T02:31:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:31:20.655 INFO:teuthology.orchestra.run.smithi017.stdout:Upgraded: 2023-07-20T02:31:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T02:31:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.655 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-07-20T02:31:20.655 INFO:teuthology.orchestra.run.smithi017.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T02:31:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T02:31:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T02:31:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.656 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.657 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.658 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:31:20.659 INFO:teuthology.orchestra.run.smithi017.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T02:31:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T02:31:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T02:31:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T02:31:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: jq-1.6-3.el8.x86_64 2023-07-20T02:31:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T02:31:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T02:31:20.660 INFO:teuthology.orchestra.run.smithi017.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T02:31:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T02:31:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:31:20.661 INFO:teuthology.orchestra.run.smithi017.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T02:31:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T02:31:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:31:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T02:31:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.662 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T02:31:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T02:31:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T02:31:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T02:31:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T02:31:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T02:31:20.663 INFO:teuthology.orchestra.run.smithi017.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T02:31:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T02:31:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T02:31:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T02:31:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T02:31:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T02:31:20.664 INFO:teuthology.orchestra.run.smithi017.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T02:31:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T02:31:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T02:31:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:31:20.665 INFO:teuthology.orchestra.run.smithi017.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:31:20.669 INFO:teuthology.orchestra.run.smithi017.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:31:20.669 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T02:31:20.670 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T02:31:20.670 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T02:31:20.670 INFO:teuthology.orchestra.run.smithi017.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T02:31:20.670 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T02:31:20.671 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.671 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.671 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.672 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T02:31:20.672 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T02:31:20.672 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T02:31:20.673 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T02:31:20.673 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T02:31:20.674 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T02:31:20.674 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T02:31:20.674 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T02:31:20.674 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T02:31:20.674 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T02:31:20.675 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T02:31:20.675 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T02:31:20.675 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T02:31:20.675 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T02:31:20.675 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T02:31:20.675 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T02:31:20.675 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T02:31:20.676 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T02:31:20.676 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T02:31:20.676 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T02:31:20.676 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T02:31:20.676 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T02:31:20.676 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T02:31:20.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T02:31:20.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T02:31:20.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T02:31:20.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T02:31:20.677 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T02:31:20.678 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T02:31:20.678 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.678 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.678 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T02:31:20.678 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T02:31:20.679 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.679 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T02:31:20.679 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T02:31:20.679 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T02:31:20.679 INFO:teuthology.orchestra.run.smithi017.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T02:31:20.679 INFO:teuthology.orchestra.run.smithi017.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T02:31:20.679 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T02:31:20.680 INFO:teuthology.orchestra.run.smithi017.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T02:31:20.680 INFO:teuthology.orchestra.run.smithi017.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T02:31:20.680 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T02:31:20.680 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T02:31:20.681 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T02:31:20.681 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T02:31:20.681 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T02:31:20.682 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.682 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:31:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T02:31:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T02:31:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T02:31:20.684 INFO:teuthology.orchestra.run.smithi017.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T02:31:20.684 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T02:31:20.686 INFO:teuthology.orchestra.run.smithi017.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T02:31:20.686 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:31:20.686 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-07-20T02:31:20.851 DEBUG:teuthology.parallel:result is None 2023-07-20T02:31:27.518 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:31:27.518 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2023-07-20T02:31:27.519 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2023-07-20T02:31:27.519 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /dev 2023-07-20T02:31:27.519 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /run 2023-07-20T02:31:27.519 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2023-07-20T02:31:27.520 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2023-07-20T02:31:27.520 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2023-07-20T02:31:27.520 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2023-07-20T02:31:27.521 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /dev 2023-07-20T02:31:27.521 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:31:27.908 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:31:29.486 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:31:29.671 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:31:29.889 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:31:30.067 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:31:30.520 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:31:30.725 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:31:30.742 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:31:32.377 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:31:32.548 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:31:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T02:31:32.916 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T02:31:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:31:33.356 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:31:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:31:34.618 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:31:34.815 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:31:34.908 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:31:35.315 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T02:31:35.564 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:31:35.599 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:31:35.749 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:31:35.760 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:31:37.042 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T02:31:37.257 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:31:37.291 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:31:37.440 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:31:37.483 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:31:37.690 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:31:38.042 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:31:38.207 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:31:42.766 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:31:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T02:31:43.298 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T02:31:43.498 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T02:31:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-20T02:31:44.737 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-20T02:31:44.934 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 119/134 2023-07-20T02:31:45.110 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T02:31:45.294 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T02:31:45.432 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:31:47.621 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:47.873 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:47.890 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:31:48.056 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T02:31:48.422 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T02:31:48.725 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T02:31:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T02:31:49.164 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T02:31:49.173 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T02:31:49.371 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-20T02:31:49.555 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T02:31:49.644 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T02:31:49.847 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T02:31:49.860 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T02:31:49.861 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:50.109 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:50.109 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:31:50.361 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:31:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T02:31:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T02:31:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T02:31:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T02:31:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T02:31:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:31:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:31:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:31:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:31:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T02:31:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T02:31:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T02:31:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T02:31:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T02:31:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T02:31:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T02:31:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T02:31:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T02:31:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T02:31:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T02:31:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T02:31:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T02:31:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T02:31:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T02:31:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T02:31:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T02:31:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T02:31:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T02:31:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T02:31:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T02:31:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T02:31:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T02:31:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T02:31:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T02:31:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T02:31:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T02:31:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T02:31:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T02:31:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T02:31:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T02:31:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T02:31:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T02:31:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T02:31:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-20T02:31:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-20T02:31:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-20T02:31:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-20T02:31:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-20T02:31:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-20T02:31:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-20T02:31:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-20T02:31:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-20T02:31:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-20T02:31:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-20T02:31:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-20T02:31:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-20T02:31:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T02:31:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-20T02:31:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:31:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-20T02:31:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-20T02:31:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-20T02:31:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-20T02:31:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-20T02:31:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T02:31:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T02:31:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T02:31:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T02:31:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:31:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T02:31:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:31:53.645 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:57 ago on Thu 20 Jul 2023 02:29:56 AM UTC. 2023-07-20T02:32:01.593 INFO:teuthology.orchestra.run.smithi151.stdout:Installed products updated. 2023-07-20T02:32:01.810 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:32:01.811 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2023-07-20T02:32:01.811 INFO:teuthology.orchestra.run.smithi151.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T02:32:01.811 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.811 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.811 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-07-20T02:32:01.813 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T02:32:01.813 INFO:teuthology.orchestra.run.smithi151.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T02:32:01.814 INFO:teuthology.orchestra.run.smithi151.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T02:32:01.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.815 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.815 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.815 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.815 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.815 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.815 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.815 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.816 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.816 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.816 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.816 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.816 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.816 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.816 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.817 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.817 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.817 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:32:01.817 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T02:32:01.817 INFO:teuthology.orchestra.run.smithi151.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T02:32:01.817 INFO:teuthology.orchestra.run.smithi151.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T02:32:01.817 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T02:32:01.818 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.6-3.el8.x86_64 2023-07-20T02:32:01.818 INFO:teuthology.orchestra.run.smithi151.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T02:32:01.818 INFO:teuthology.orchestra.run.smithi151.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T02:32:01.818 INFO:teuthology.orchestra.run.smithi151.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T02:32:01.818 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.818 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.818 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.819 INFO:teuthology.orchestra.run.smithi151.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T02:32:01.819 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:32:01.819 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T02:32:01.819 INFO:teuthology.orchestra.run.smithi151.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T02:32:01.819 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:32:01.820 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T02:32:01.820 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.820 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.820 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T02:32:01.820 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.820 INFO:teuthology.orchestra.run.smithi151.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T02:32:01.820 INFO:teuthology.orchestra.run.smithi151.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T02:32:01.821 INFO:teuthology.orchestra.run.smithi151.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T02:32:01.821 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T02:32:01.821 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T02:32:01.821 INFO:teuthology.orchestra.run.smithi151.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T02:32:01.821 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T02:32:01.821 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T02:32:01.822 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T02:32:01.822 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T02:32:01.822 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T02:32:01.822 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T02:32:01.822 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T02:32:01.822 INFO:teuthology.orchestra.run.smithi151.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T02:32:01.823 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:32:01.823 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:32:01.823 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:32:01.824 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T02:32:01.824 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T02:32:01.824 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T02:32:01.824 INFO:teuthology.orchestra.run.smithi151.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T02:32:01.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T02:32:01.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T02:32:01.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T02:32:01.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T02:32:01.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T02:32:01.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T02:32:01.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T02:32:01.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T02:32:01.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T02:32:01.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T02:32:01.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T02:32:01.827 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T02:32:01.828 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T02:32:01.828 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T02:32:01.828 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T02:32:01.828 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T02:32:01.828 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T02:32:01.828 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T02:32:01.828 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T02:32:01.829 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T02:32:01.830 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.830 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T02:32:01.830 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T02:32:01.830 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T02:32:01.830 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T02:32:01.831 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T02:32:01.832 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T02:32:01.832 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.832 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.832 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:32:01.832 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T02:32:01.832 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T02:32:01.832 INFO:teuthology.orchestra.run.smithi151.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T02:32:01.833 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T02:32:01.833 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T02:32:01.833 INFO:teuthology.orchestra.run.smithi151.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T02:32:01.833 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:32:01.834 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-07-20T02:32:02.004 DEBUG:teuthology.parallel:result is None 2023-07-20T02:32:02.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:32:02.436 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T02:32:02.467 INFO:teuthology.orchestra.run.smithi017.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:32:02.468 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:32:02.468 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T02:32:02.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:32:02.910 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T02:32:02.941 INFO:teuthology.orchestra.run.smithi146.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:32:02.942 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:32:02.942 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T02:32:02.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:32:03.469 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T02:32:03.497 INFO:teuthology.orchestra.run.smithi151.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:32:03.498 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:32:03.498 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T02:32:03.498 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-20T02:32:03.499 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:32:03.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T02:32:03.549 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:32:03.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T02:32:03.598 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:32:03.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T02:32:03.647 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-20T02:32:03.648 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:32:03.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T02:32:03.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T02:32:03.750 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:32:03.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T02:32:03.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T02:32:03.852 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:32:03.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T02:32:03.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T02:32:03.957 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-20T02:32:03.957 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:32:03.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T02:32:03.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T02:32:04.065 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:32:04.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T02:32:04.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T02:32:04.167 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:32:04.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T02:32:04.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T02:32:04.270 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-20T02:32:04.271 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:32:04.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T02:32:04.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T02:32:04.377 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:32:04.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T02:32:04.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T02:32:04.479 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:32:04.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T02:32:04.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T02:32:04.580 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-20T02:32:04.676 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'subvols': {'create': 2}} 2023-07-20T02:32:04.677 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:32:04.687 INFO:tasks.cephadm:Cluster fsid is 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:04.688 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-20T02:32:04.689 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.17', 'mon.b': '172.21.15.146', 'mon.c': '172.21.15.151'} 2023-07-20T02:32:04.689 INFO:tasks.cephadm:First mon is mon.a on smithi017 2023-07-20T02:32:04.689 INFO:tasks.cephadm:First mgr is x 2023-07-20T02:32:04.689 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-20T02:32:04.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo hostname $(hostname -s) 2023-07-20T02:32:04.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2023-07-20T02:32:04.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2023-07-20T02:32:04.789 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-20T02:32:04.790 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:32:05.282 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71562/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5040-g1b0b2eb0', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5040.g1b0b2eb0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-19 22:37:13.930734', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230719.201220', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-20T02:32:05.872 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-pdonnell-testing-20230719.201220, sha1 1b0b2eb005a7c51d97a788b9900d511e1394b537 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:32:05.873 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm 2023-07-20T02:32:05.873 DEBUG:teuthology.orchestra.run.smithi017:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:07.583 INFO:teuthology.orchestra.run.smithi017.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 02:32 /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:07.583 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:09.089 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 02:32 /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:09.090 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:10.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 02:32 /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:10.429 DEBUG:teuthology.orchestra.run.smithi017:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:10.451 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:10.473 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T02:32:10.510 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 on all hosts... 2023-07-20T02:32:10.511 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T02:32:10.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T02:32:10.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T02:32:11.769 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:32:11.958 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:32:14.015 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:32:39.249 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2023-07-20T02:32:39.249 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T02:32:39.250 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T02:32:39.250 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2023-07-20T02:32:39.250 INFO:teuthology.orchestra.run.smithi146.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T02:32:39.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2023-07-20T02:32:39.251 INFO:teuthology.orchestra.run.smithi146.stdout:} 2023-07-20T02:32:41.763 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-07-20T02:32:41.764 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T02:32:41.764 INFO:teuthology.orchestra.run.smithi017.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T02:32:41.764 INFO:teuthology.orchestra.run.smithi017.stdout: "repo_digests": [ 2023-07-20T02:32:41.764 INFO:teuthology.orchestra.run.smithi017.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T02:32:41.765 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2023-07-20T02:32:41.765 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-07-20T02:32:42.508 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-07-20T02:32:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T02:32:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T02:32:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2023-07-20T02:32:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T02:32:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2023-07-20T02:32:42.509 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-07-20T02:32:42.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph 2023-07-20T02:32:42.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2023-07-20T02:32:42.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2023-07-20T02:32:42.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /etc/ceph 2023-07-20T02:32:42.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2023-07-20T02:32:42.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2023-07-20T02:32:42.719 INFO:tasks.cephadm:Writing seed config... 2023-07-20T02:32:42.720 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-20T02:32:42.721 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-20T02:32:42.721 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-20T02:32:42.721 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-20T02:32:42.721 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-20T02:32:42.721 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-20T02:32:42.722 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-20T02:32:42.722 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-20T02:32:42.722 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-20T02:32:42.722 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-20T02:32:42.722 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-20T02:32:42.723 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-20T02:32:42.723 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-20T02:32:42.723 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-20T02:32:42.723 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-20T02:32:42.723 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-20T02:32:42.724 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-20T02:32:42.724 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-07-20T02:32:42.724 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-20T02:32:42.724 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-20T02:32:42.724 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-20T02:32:42.725 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-20T02:32:42.725 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-20T02:32:42.725 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-20T02:32:42.725 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-20T02:32:42.725 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-20T02:32:42.726 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-20T02:32:42.726 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-20T02:32:42.726 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-20T02:32:42.726 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-07-20T02:32:42.726 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-07-20T02:32:42.727 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-07-20T02:32:42.727 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-20T02:32:42.727 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-20T02:32:42.727 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-07-20T02:32:42.727 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-07-20T02:32:42.728 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-20T02:32:42.728 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-20T02:32:42.728 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-07-20T02:32:42.728 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-20T02:32:42.728 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-20T02:32:42.729 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-20T02:32:42.729 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-20T02:32:42.729 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-20T02:32:42.729 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-20T02:32:42.729 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-20T02:32:42.729 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-20T02:32:42.730 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:32:42.731 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-20T02:32:42.750 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9d6085e8-26a5-11ee-9b34-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-20T02:32:42.751 DEBUG:teuthology.orchestra.run.smithi017:mon.a> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a.service 2023-07-20T02:32:42.792 DEBUG:teuthology.orchestra.run.smithi017:mgr.x> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.x.service 2023-07-20T02:32:42.824 INFO:journalctl@ceph.mon.a.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:32:42.825 INFO:tasks.cephadm:Bootstrapping... 2023-07-20T02:32:42.825 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 -v bootstrap --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.17 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:32:42.854 INFO:journalctl@ceph.mgr.x.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:32:42.997 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:32:42.997 INFO:teuthology.orchestra.run.smithi017.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537', '-v', 'bootstrap', '--fsid', '9d6085e8-26a5-11ee-9b34-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.17', '--skip-admin-label'] 2023-07-20T02:32:43.142 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stdout 4.1.1 2023-07-20T02:32:43.142 INFO:teuthology.orchestra.run.smithi017.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-20T02:32:43.142 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying podman|docker is present... 2023-07-20T02:32:43.290 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stdout 4.1.1 2023-07-20T02:32:43.290 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying lvm2 is present... 2023-07-20T02:32:43.290 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying time synchronization is in place... 2023-07-20T02:32:43.297 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T02:32:43.297 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T02:32:43.303 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T02:32:43.303 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2023-07-20T02:32:43.309 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout enabled 2023-07-20T02:32:43.315 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout active 2023-07-20T02:32:43.315 INFO:teuthology.orchestra.run.smithi017.stdout:Unit chronyd.service is enabled and running 2023-07-20T02:32:43.316 INFO:teuthology.orchestra.run.smithi017.stdout:Repeating the final host check... 2023-07-20T02:32:43.461 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stdout 4.1.1 2023-07-20T02:32:43.461 INFO:teuthology.orchestra.run.smithi017.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-20T02:32:43.461 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl is present 2023-07-20T02:32:43.461 INFO:teuthology.orchestra.run.smithi017.stdout:lvcreate is present 2023-07-20T02:32:43.467 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T02:32:43.468 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T02:32:43.473 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T02:32:43.473 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2023-07-20T02:32:43.479 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout enabled 2023-07-20T02:32:43.485 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout active 2023-07-20T02:32:43.485 INFO:teuthology.orchestra.run.smithi017.stdout:Unit chronyd.service is enabled and running 2023-07-20T02:32:43.485 INFO:teuthology.orchestra.run.smithi017.stdout:Host looks OK 2023-07-20T02:32:43.485 INFO:teuthology.orchestra.run.smithi017.stdout:Cluster fsid: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:43.485 INFO:teuthology.orchestra.run.smithi017.stdout:Acquiring lock 140717260511720 on /run/cephadm/9d6085e8-26a5-11ee-9b34-001a4aab830c.lock 2023-07-20T02:32:43.486 INFO:teuthology.orchestra.run.smithi017.stdout:Lock 140717260511720 acquired on /run/cephadm/9d6085e8-26a5-11ee-9b34-001a4aab830c.lock 2023-07-20T02:32:43.487 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying IP 172.21.15.17 port 3300 ... 2023-07-20T02:32:43.487 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying IP 172.21.15.17 port 6789 ... 2023-07-20T02:32:43.487 INFO:teuthology.orchestra.run.smithi017.stdout:Base mon IP(s) is [172.21.15.17:3300, 172.21.15.17:6789], mon addrv is [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2023-07-20T02:32:43.490 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-20T02:32:43.490 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.17 metric 100 2023-07-20T02:32:43.492 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-20T02:32:43.492 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-20T02:32:43.492 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2023-07-20T02:32:43.495 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-20T02:32:43.495 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-20T02:32:43.495 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T02:32:43.495 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-20T02:32:43.495 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:d992/64 scope link 2023-07-20T02:32:43.495 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T02:32:43.496 INFO:teuthology.orchestra.run.smithi017.stdout:Mon IP `172.21.15.17` is in CIDR network `172.21.0.0/20` 2023-07-20T02:32:43.496 INFO:teuthology.orchestra.run.smithi017.stdout:Mon IP `172.21.15.17` is in CIDR network `172.21.0.0/20` 2023-07-20T02:32:43.496 INFO:teuthology.orchestra.run.smithi017.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-20T02:32:43.497 INFO:teuthology.orchestra.run.smithi017.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-20T02:32:43.497 INFO:teuthology.orchestra.run.smithi017.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:32:44.053 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stdout 3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T02:32:44.054 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:32:44.054 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Getting image source signatures 2023-07-20T02:32:44.054 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Copying blob sha256:9a3af2d1f966fade87cf08b715899d7eff2dde16dabc6e2e4c3ec2059da8adc1 2023-07-20T02:32:44.055 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-20T02:32:44.055 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Copying config sha256:3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T02:32:44.055 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-20T02:32:44.056 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Storing signatures 2023-07-20T02:32:44.907 INFO:teuthology.orchestra.run.smithi017.stdout:ceph: stdout ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T02:32:44.907 INFO:teuthology.orchestra.run.smithi017.stdout:Ceph version: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T02:32:44.907 INFO:teuthology.orchestra.run.smithi017.stdout:Extracting ceph user uid/gid from container image... 2023-07-20T02:32:45.560 INFO:teuthology.orchestra.run.smithi017.stdout:stat: stdout 167 167 2023-07-20T02:32:45.560 INFO:teuthology.orchestra.run.smithi017.stdout:Creating initial keys... 2023-07-20T02:32:46.316 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph-authtool: stdout AQDNnLhkbR92NxAALBkLL90q7tUFJqRTpBbesw== 2023-07-20T02:32:47.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph-authtool: stdout AQDOnLhk5YYQKRAAy4z2/V8oVfrw+D9cc1cZYw== 2023-07-20T02:32:47.712 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph-authtool: stdout AQDPnLhkvuzVGBAA/zyB2dK4VeNHgwaSXDeZIw== 2023-07-20T02:32:47.713 INFO:teuthology.orchestra.run.smithi017.stdout:Creating initial monmap... 2023-07-20T02:32:48.363 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T02:32:48.363 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-20T02:32:48.363 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:48.363 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T02:32:48.363 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool for a [v2:172.21.15.17:3300,v1:172.21.15.17:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T02:32:48.363 INFO:teuthology.orchestra.run.smithi017.stdout:setting min_mon_release = pacific 2023-07-20T02:32:48.364 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: set fsid to 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:48.364 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T02:32:48.364 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:32:48.364 INFO:teuthology.orchestra.run.smithi017.stdout:Creating mon... 2023-07-20T02:32:49.009 INFO:teuthology.orchestra.run.smithi017.stdout:create mon.a on 2023-07-20T02:32:49.151 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-20T02:32:49.242 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-20T02:32:49.362 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c.target. 2023-07-20T02:32:49.363 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c.target. 2023-07-20T02:32:49.547 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a 2023-07-20T02:32:49.547 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a.service: Unit ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a.service not loaded. 2023-07-20T02:32:49.636 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c.target.wants/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@.service. 2023-07-20T02:32:49.910 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:49 smithi017 systemd[1]: Starting Ceph mon.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:32:50.144 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:32:50.144 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2023-07-20T02:32:50.152 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:32:50.153 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2023-07-20T02:32:50.153 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2023-07-20T02:32:50.153 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T02:32:50.153 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mon to start... 2023-07-20T02:32:50.154 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mon... 2023-07-20T02:32:50.166 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:49 smithi017 podman[126809]: 2023-07-20T02:32:50.166 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:50 smithi017 bash[126809]: b296ce5544a38dfa695b8105a8f2431367cffecc8dc05d5728b9e2cda462bbd3 2023-07-20T02:32:50.166 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:50 smithi017 systemd[1]: Started Ceph mon.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:32:50.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:50 smithi017 ceph-mon[126850]: mkfs 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:50.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:50 smithi017 ceph-mon[126850]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T02:32:51.109 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout cluster: 2023-07-20T02:32:51.109 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout id: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:51.109 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-20T02:32:51.109 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:32:51.110 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout services: 2023-07-20T02:32:51.110 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.600189s) 2023-07-20T02:32:51.110 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-20T02:32:51.110 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-20T02:32:51.110 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:32:51.110 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout data: 2023-07-20T02:32:51.110 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout pgs: 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.635+0000 7f7a8edba700 1 Processor -- start 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.635+0000 7f7a8edba700 1 -- start start 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.635+0000 7f7a8edba700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88104890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:51.111 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.635+0000 7f7a8edba700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88104e60 con 0x7f7a88104470 2023-07-20T02:32:51.112 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a8cb56700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88104890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:51.112 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a8cb56700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88104890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42256/0 (socket says 172.21.15.17:42256) 2023-07-20T02:32:51.112 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a8cb56700 1 -- 172.21.15.17:0/4051224795 learned_addr learned my addr 172.21.15.17:0/4051224795 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:51.112 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a8cb56700 1 -- 172.21.15.17:0/4051224795 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a88107400 con 0x7f7a88104470 2023-07-20T02:32:51.112 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a8cb56700 1 --2- 172.21.15.17:0/4051224795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88104890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0098a0 tx=0x7f7a7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa52d6231f888b04 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:51.112 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a877fe700 1 -- 172.21.15.17:0/4051224795 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a7c004160 con 0x7f7a88104470 2023-07-20T02:32:51.113 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a877fe700 1 -- 172.21.15.17:0/4051224795 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c0042c0 con 0x7f7a88104470 2023-07-20T02:32:51.113 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.636+0000 7f7a877fe700 1 -- 172.21.15.17:0/4051224795 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a7c0045a0 con 0x7f7a88104470 2023-07-20T02:32:51.113 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.637+0000 7f7a8edba700 1 -- 172.21.15.17:0/4051224795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 msgr2=0x7f7a88104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:51.113 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.637+0000 7f7a8edba700 1 --2- 172.21.15.17:0/4051224795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88104890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0098a0 tx=0x7f7a7c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:32:51.113 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.637+0000 7f7a8edba700 1 -- 172.21.15.17:0/4051224795 shutdown_connections 2023-07-20T02:32:51.113 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.637+0000 7f7a8edba700 1 --2- 172.21.15.17:0/4051224795 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88104890 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:51.114 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.637+0000 7f7a8edba700 1 -- 172.21.15.17:0/4051224795 >> 172.21.15.17:0/4051224795 conn(0x7f7a880ffdc0 msgr2=0x7f7a881021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:51.114 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.637+0000 7f7a8edba700 1 -- 172.21.15.17:0/4051224795 shutdown_connections 2023-07-20T02:32:51.114 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.637+0000 7f7a8edba700 1 -- 172.21.15.17:0/4051224795 wait complete. 2023-07-20T02:32:51.114 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8edba700 1 Processor -- start 2023-07-20T02:32:51.114 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8edba700 1 -- start start 2023-07-20T02:32:51.114 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8edba700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88106d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:51.114 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8edba700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881072a0 con 0x7f7a88104470 2023-07-20T02:32:51.115 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8cb56700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88106d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:51.115 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8cb56700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88106d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42258/0 (socket says 172.21.15.17:42258) 2023-07-20T02:32:51.115 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8cb56700 1 -- 172.21.15.17:0/2622604069 learned_addr learned my addr 172.21.15.17:0/2622604069 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:51.115 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.638+0000 7f7a8cb56700 1 -- 172.21.15.17:0/2622604069 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c009550 con 0x7f7a88104470 2023-07-20T02:32:51.115 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a8cb56700 1 --2- 172.21.15.17:0/2622604069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88106d60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c004000 tx=0x7f7a7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:51.115 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a85ffb700 1 -- 172.21.15.17:0/2622604069 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a7c011630 con 0x7f7a88104470 2023-07-20T02:32:51.115 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a85ffb700 1 -- 172.21.15.17:0/2622604069 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c011790 con 0x7f7a88104470 2023-07-20T02:32:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a881053b0 con 0x7f7a88104470 2023-07-20T02:32:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a85ffb700 1 -- 172.21.15.17:0/2622604069 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a7c011a30 con 0x7f7a88104470 2023-07-20T02:32:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a881057d0 con 0x7f7a88104470 2023-07-20T02:32:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a85ffb700 1 -- 172.21.15.17:0/2622604069 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7a7c022070 con 0x7f7a88104470 2023-07-20T02:32:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.639+0000 7f7a85ffb700 1 -- 172.21.15.17:0/2622604069 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7a7c01c450 con 0x7f7a88104470 2023-07-20T02:32:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.640+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a881061e0 con 0x7f7a88104470 2023-07-20T02:32:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.642+0000 7f7a85ffb700 1 -- 172.21.15.17:0/2622604069 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7a7c01b450 con 0x7f7a88104470 2023-07-20T02:32:51.117 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.773+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7a88192a10 con 0x7f7a88104470 2023-07-20T02:32:51.117 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.773+0000 7f7a85ffb700 1 -- 172.21.15.17:0/2622604069 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f7a7c01b5f0 con 0x7f7a88104470 2023-07-20T02:32:51.117 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.774+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 msgr2=0x7f7a88106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:51.117 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.774+0000 7f7a8edba700 1 --2- 172.21.15.17:0/2622604069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88106d60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c004000 tx=0x7f7a7c004750 comp rx=0 tx=0).stop 2023-07-20T02:32:51.117 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.774+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 shutdown_connections 2023-07-20T02:32:51.117 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.774+0000 7f7a8edba700 1 --2- 172.21.15.17:0/2622604069 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7a88104470 0x7f7a88106d60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:51.117 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.774+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 >> 172.21.15.17:0/2622604069 conn(0x7f7a880ffdc0 msgr2=0x7f7a8818dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:51.118 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.774+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 shutdown_connections 2023-07-20T02:32:51.118 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:50.774+0000 7f7a8edba700 1 -- 172.21.15.17:0/2622604069 wait complete. 2023-07-20T02:32:51.118 INFO:teuthology.orchestra.run.smithi017.stdout:mon is available 2023-07-20T02:32:51.118 INFO:teuthology.orchestra.run.smithi017.stdout:Assimilating anything we can from ceph.conf... 2023-07-20T02:32:51.334 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:51 smithi017 ceph-mon[126850]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T02:32:51.334 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:51 smithi017 ceph-mon[126850]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} removed_ranks: {} 2023-07-20T02:32:51.334 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:51 smithi017 ceph-mon[126850]: fsmap 2023-07-20T02:32:51.334 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:51 smithi017 ceph-mon[126850]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T02:32:51.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:51 smithi017 ceph-mon[126850]: mgrmap e1: no daemons active 2023-07-20T02:32:51.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:51 smithi017 ceph-mon[126850]: from='client.? 172.21.15.17:0/2622604069' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-20T02:32:52.055 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:32:52.055 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T02:32:52.056 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout fsid = 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:52.056 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2023-07-20T02:32:52.056 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T02:32:52.056 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T02:32:52.056 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T02:32:52.056 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T02:32:52.056 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:32:52.057 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T02:32:52.057 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T02:32:52.057 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:32:52.057 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T02:32:52.057 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T02:32:52.057 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T02:32:52.057 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T02:32:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T02:32:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T02:32:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.562+0000 7fde25665700 1 Processor -- start 2023-07-20T02:32:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.562+0000 7fde25665700 1 -- start start 2023-07-20T02:32:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.562+0000 7fde25665700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde20109060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.562+0000 7fde25665700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde20073fb0 con 0x7fde20106c70 2023-07-20T02:32:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.563+0000 7fde1effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde20109060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:52.059 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.563+0000 7fde1effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde20109060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42260/0 (socket says 172.21.15.17:42260) 2023-07-20T02:32:52.059 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.563+0000 7fde1effd700 1 -- 172.21.15.17:0/1976255749 learned_addr learned my addr 172.21.15.17:0/1976255749 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:52.059 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.563+0000 7fde1effd700 1 -- 172.21.15.17:0/1976255749 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde201095a0 con 0x7fde20106c70 2023-07-20T02:32:52.059 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.563+0000 7fde1effd700 1 --2- 172.21.15.17:0/1976255749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde20109060 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fde100098a0 tx=0x7fde10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf39d19c7d3ced server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:52.059 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde1dffb700 1 -- 172.21.15.17:0/1976255749 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde10004160 con 0x7fde20106c70 2023-07-20T02:32:52.060 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde1dffb700 1 -- 172.21.15.17:0/1976255749 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde100042c0 con 0x7fde20106c70 2023-07-20T02:32:52.060 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde1dffb700 1 -- 172.21.15.17:0/1976255749 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde100045a0 con 0x7fde20106c70 2023-07-20T02:32:52.060 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde25665700 1 -- 172.21.15.17:0/1976255749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 msgr2=0x7fde20109060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:52.060 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde25665700 1 --2- 172.21.15.17:0/1976255749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde20109060 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fde100098a0 tx=0x7fde10009bb0 comp rx=0 tx=0).stop 2023-07-20T02:32:52.060 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde25665700 1 -- 172.21.15.17:0/1976255749 shutdown_connections 2023-07-20T02:32:52.060 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde25665700 1 --2- 172.21.15.17:0/1976255749 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde20109060 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:52.060 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde25665700 1 -- 172.21.15.17:0/1976255749 >> 172.21.15.17:0/1976255749 conn(0x7fde20100680 msgr2=0x7fde20102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde25665700 1 -- 172.21.15.17:0/1976255749 shutdown_connections 2023-07-20T02:32:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.564+0000 7fde25665700 1 -- 172.21.15.17:0/1976255749 wait complete. 2023-07-20T02:32:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.565+0000 7fde25665700 1 Processor -- start 2023-07-20T02:32:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.565+0000 7fde25665700 1 -- start start 2023-07-20T02:32:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.565+0000 7fde25665700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde2019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.565+0000 7fde25665700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2019bdc0 con 0x7fde20106c70 2023-07-20T02:32:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.565+0000 7fde1effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde2019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:52.062 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.565+0000 7fde1effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde2019b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42262/0 (socket says 172.21.15.17:42262) 2023-07-20T02:32:52.062 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.565+0000 7fde1effd700 1 -- 172.21.15.17:0/1446747844 learned_addr learned my addr 172.21.15.17:0/1446747844 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:52.062 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde1effd700 1 -- 172.21.15.17:0/1446747844 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde10009550 con 0x7fde20106c70 2023-07-20T02:32:52.062 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde1effd700 1 --2- 172.21.15.17:0/1446747844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde2019b880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fde200733f0 tx=0x7fde10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:52.062 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde10011630 con 0x7fde20106c70 2023-07-20T02:32:52.062 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde10011790 con 0x7fde20106c70 2023-07-20T02:32:52.063 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2019bfc0 con 0x7fde20106c70 2023-07-20T02:32:52.063 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde10011a30 con 0x7fde20106c70 2023-07-20T02:32:52.063 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2019c3e0 con 0x7fde20106c70 2023-07-20T02:32:52.063 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fde10022070 con 0x7fde20106c70 2023-07-20T02:32:52.063 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.566+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fde1001c450 con 0x7fde20106c70 2023-07-20T02:32:52.063 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.567+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde20195900 con 0x7fde20106c70 2023-07-20T02:32:52.063 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.570+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fde1001b450 con 0x7fde20106c70 2023-07-20T02:32:52.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.697+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fde2002cc20 con 0x7fde20106c70 2023-07-20T02:32:52.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.700+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde1001b5f0 con 0x7fde20106c70 2023-07-20T02:32:52.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.700+0000 7fde17fff700 1 -- 172.21.15.17:0/1446747844 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fde10024630 con 0x7fde20106c70 2023-07-20T02:32:52.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.701+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 msgr2=0x7fde2019b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:52.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.701+0000 7fde25665700 1 --2- 172.21.15.17:0/1446747844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde2019b880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fde200733f0 tx=0x7fde10004750 comp rx=0 tx=0).stop 2023-07-20T02:32:52.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.701+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 shutdown_connections 2023-07-20T02:32:52.064 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.701+0000 7fde25665700 1 --2- 172.21.15.17:0/1446747844 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fde20106c70 0x7fde2019b880 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:52.065 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.701+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 >> 172.21.15.17:0/1446747844 conn(0x7fde20100680 msgr2=0x7fde20101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:52.065 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.701+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 shutdown_connections 2023-07-20T02:32:52.065 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:51.702+0000 7fde25665700 1 -- 172.21.15.17:0/1446747844 wait complete. 2023-07-20T02:32:52.065 INFO:teuthology.orchestra.run.smithi017.stdout:Generating new minimal ceph.conf... 2023-07-20T02:32:52.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:52 smithi017 ceph-mon[126850]: from='client.? 172.21.15.17:0/1446747844' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T02:32:52.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:52 smithi017 ceph-mon[126850]: from='client.? 172.21.15.17:0/1446747844' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-20T02:32:53.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.520+0000 7f8bfb1c2700 1 Processor -- start 2023-07-20T02:32:53.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.520+0000 7f8bfb1c2700 1 -- start start 2023-07-20T02:32:53.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bfb1c2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf41090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:53.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bfb1c2700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4073f70 con 0x7f8bf4106cb0 2023-07-20T02:32:53.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf8f5e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf41090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:53.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf8f5e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf41090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42264/0 (socket says 172.21.15.17:42264) 2023-07-20T02:32:53.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf8f5e700 1 -- 172.21.15.17:0/2911388576 learned_addr learned my addr 172.21.15.17:0/2911388576 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:53.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf8f5e700 1 -- 172.21.15.17:0/2911388576 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf41095e0 con 0x7f8bf4106cb0 2023-07-20T02:32:53.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf8f5e700 1 --2- 172.21.15.17:0/2911388576 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf41090a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8be80098a0 tx=0x7f8be8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aede27aa3a8d63e server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:53.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf37fe700 1 -- 172.21.15.17:0/2911388576 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be8004160 con 0x7f8bf4106cb0 2023-07-20T02:32:53.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf37fe700 1 -- 172.21.15.17:0/2911388576 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8be80042c0 con 0x7f8bf4106cb0 2023-07-20T02:32:53.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.521+0000 7f8bf37fe700 1 -- 172.21.15.17:0/2911388576 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be8011a00 con 0x7f8bf4106cb0 2023-07-20T02:32:53.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.522+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/2911388576 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 msgr2=0x7f8bf41090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:53.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.522+0000 7f8bfb1c2700 1 --2- 172.21.15.17:0/2911388576 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf41090a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8be80098a0 tx=0x7f8be8009bb0 comp rx=0 tx=0).stop 2023-07-20T02:32:53.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.522+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/2911388576 shutdown_connections 2023-07-20T02:32:53.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.522+0000 7f8bfb1c2700 1 --2- 172.21.15.17:0/2911388576 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf41090a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:53.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.522+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/2911388576 >> 172.21.15.17:0/2911388576 conn(0x7f8bf4100680 msgr2=0x7f8bf4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:53.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.522+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/2911388576 shutdown_connections 2023-07-20T02:32:53.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.522+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/2911388576 wait complete. 2023-07-20T02:32:53.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bfb1c2700 1 Processor -- start 2023-07-20T02:32:53.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bfb1c2700 1 -- start start 2023-07-20T02:32:53.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bfb1c2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf4199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:53.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bfb1c2700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf419a480 con 0x7f8bf4106cb0 2023-07-20T02:32:53.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bf8f5e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf4199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:53.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bf8f5e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf4199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42266/0 (socket says 172.21.15.17:42266) 2023-07-20T02:32:53.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bf8f5e700 1 -- 172.21.15.17:0/1485544234 learned_addr learned my addr 172.21.15.17:0/1485544234 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:53.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.523+0000 7f8bf8f5e700 1 -- 172.21.15.17:0/1485544234 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be8009550 con 0x7f8bf4106cb0 2023-07-20T02:32:53.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bf8f5e700 1 --2- 172.21.15.17:0/1485544234 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf4199f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009130 tx=0x7f8be8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:53.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bf1ffb700 1 -- 172.21.15.17:0/1485544234 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be8003b70 con 0x7f8bf4106cb0 2023-07-20T02:32:53.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bf1ffb700 1 -- 172.21.15.17:0/1485544234 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8be8003cd0 con 0x7f8bf4106cb0 2023-07-20T02:32:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf419a680 con 0x7f8bf4106cb0 2023-07-20T02:32:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bf1ffb700 1 -- 172.21.15.17:0/1485544234 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be8019780 con 0x7f8bf4106cb0 2023-07-20T02:32:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf419aaa0 con 0x7f8bf4106cb0 2023-07-20T02:32:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bf1ffb700 1 -- 172.21.15.17:0/1485544234 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8be8019970 con 0x7f8bf4106cb0 2023-07-20T02:32:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.524+0000 7f8bf1ffb700 1 -- 172.21.15.17:0/1485544234 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8be800b720 con 0x7f8bf4106cb0 2023-07-20T02:32:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.525+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4061da0 con 0x7f8bf4106cb0 2023-07-20T02:32:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.527+0000 7f8bf1ffb700 1 -- 172.21.15.17:0/1485544234 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f8be802a060 con 0x7f8bf4106cb0 2023-07-20T02:32:53.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.654+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8bf419d640 con 0x7f8bf4106cb0 2023-07-20T02:32:53.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.654+0000 7f8bf1ffb700 1 -- 172.21.15.17:0/1485544234 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f8be8019d60 con 0x7f8bf4106cb0 2023-07-20T02:32:53.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.655+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 msgr2=0x7f8bf4199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:53.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.655+0000 7f8bfb1c2700 1 --2- 172.21.15.17:0/1485544234 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf4199f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009130 tx=0x7f8be8011ad0 comp rx=0 tx=0).stop 2023-07-20T02:32:53.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.655+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 shutdown_connections 2023-07-20T02:32:53.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.655+0000 7f8bfb1c2700 1 --2- 172.21.15.17:0/1485544234 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8bf4106cb0 0x7f8bf4199f40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:53.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.655+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 >> 172.21.15.17:0/1485544234 conn(0x7f8bf4100680 msgr2=0x7f8bf4101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:53.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.655+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 shutdown_connections 2023-07-20T02:32:53.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:52.655+0000 7f8bfb1c2700 1 -- 172.21.15.17:0/1485544234 wait complete. 2023-07-20T02:32:53.008 INFO:teuthology.orchestra.run.smithi017.stdout:Restarting the monitor... 2023-07-20T02:32:53.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 systemd[1]: Stopping Ceph mon.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:32:53.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 ceph-mon[126850]: from='client.? 172.21.15.17:0/1485544234' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:32:53.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-a[126825]: 2023-07-20T02:32:53.339+0000 7f2cb3315700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:32:53.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-a[126825]: 2023-07-20T02:32:53.339+0000 7f2cb3315700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-20T02:32:53.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 bash[127183]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-a 2023-07-20T02:32:53.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 systemd[1]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T02:32:53.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 systemd[1]: Stopped Ceph mon.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:32:53.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:53 smithi017 systemd[1]: Starting Ceph mon.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:32:54.277 INFO:teuthology.orchestra.run.smithi017.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-20T02:32:54.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 podman[127305]: 2023-07-20T02:32:54.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 bash[127305]: 3d487ff4c5dd5408078f09e9202099ce24a761f4a1daec3a4c5902a08a29b37b 2023-07-20T02:32:54.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T02:32:54.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T02:32:54.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: pidfile_write: ignore empty --pid-file 2023-07-20T02:32:54.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: load: jerasure load: lrc 2023-07-20T02:32:54.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: RocksDB version: 7.9.2 2023-07-20T02:32:54.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Git sha 0 2023-07-20T02:32:54.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T02:32:54.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: DB SUMMARY 2023-07-20T02:32:54.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: DB Session ID: FBV7HXVTR7SRL1ATPR11 2023-07-20T02:32:54.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: CURRENT file: CURRENT 2023-07-20T02:32:54.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T02:32:54.343 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-20T02:32:54.343 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-20T02:32:54.343 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100055 ; 2023-07-20T02:32:54.343 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.error_if_exists: 0 2023-07-20T02:32:54.344 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.create_if_missing: 0 2023-07-20T02:32:54.344 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.paranoid_checks: 1 2023-07-20T02:32:54.344 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T02:32:54.344 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T02:32:54.345 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T02:32:54.345 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.env: 0x5602ba205c60 2023-07-20T02:32:54.345 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T02:32:54.346 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.info_log: 0x5602bca152c0 2023-07-20T02:32:54.346 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T02:32:54.346 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.statistics: (nil) 2023-07-20T02:32:54.346 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.use_fsync: 0 2023-07-20T02:32:54.347 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_log_file_size: 0 2023-07-20T02:32:54.347 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T02:32:54.347 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T02:32:54.348 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T02:32:54.348 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T02:32:54.348 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.allow_fallocate: 1 2023-07-20T02:32:54.348 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T02:32:54.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T02:32:54.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.use_direct_reads: 0 2023-07-20T02:32:54.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T02:32:54.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T02:32:54.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.db_log_dir: 2023-07-20T02:32:54.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.wal_dir: 2023-07-20T02:32:54.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T02:32:54.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T02:32:54.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T02:32:54.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T02:32:54.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T02:32:54.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T02:32:54.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.write_buffer_manager: 0x5602bc9363c0 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T02:32:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T02:32:54.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T02:32:54.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T02:32:54.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.unordered_write: 0 2023-07-20T02:32:54.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T02:32:54.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T02:32:54.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T02:32:54.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T02:32:54.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.row_cache: None 2023-07-20T02:32:54.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.wal_filter: None 2023-07-20T02:32:54.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T02:32:54.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T02:32:54.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.two_write_queues: 0 2023-07-20T02:32:54.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T02:32:54.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.wal_compression: 0 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.atomic_flush: 0 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.log_readahead_size: 0 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T02:32:54.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T02:32:54.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T02:32:54.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T02:32:54.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T02:32:54.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T02:32:54.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_background_jobs: 2 2023-07-20T02:32:54.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_background_compactions: -1 2023-07-20T02:32:54.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_subcompactions: 1 2023-07-20T02:32:54.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T02:32:54.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T02:32:54.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T02:32:54.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T02:32:54.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T02:32:54.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T02:32:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T02:32:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T02:32:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_open_files: -1 2023-07-20T02:32:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T02:32:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T02:32:54.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T02:32:54.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T02:32:54.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_background_flushes: -1 2023-07-20T02:32:54.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Compression algorithms supported: 2023-07-20T02:32:54.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T02:32:54.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kZSTD supported: 0 2023-07-20T02:32:54.359 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kXpressCompression supported: 0 2023-07-20T02:32:54.359 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T02:32:54.359 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kZlibCompression supported: 1 2023-07-20T02:32:54.359 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kSnappyCompression supported: 1 2023-07-20T02:32:54.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kLZ4Compression supported: 1 2023-07-20T02:32:54.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: kBZip2Compression supported: 0 2023-07-20T02:32:54.361 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T02:32:54.361 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T02:32:54.361 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-20T02:32:54.361 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T02:32:54.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T02:32:54.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.merge_operator: 2023-07-20T02:32:54.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_filter: None 2023-07-20T02:32:54.363 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T02:32:54.363 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T02:32:54.363 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T02:32:54.363 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T02:32:54.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5602bca153c0) 2023-07-20T02:32:54.364 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2023-07-20T02:32:54.364 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T02:32:54.364 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T02:32:54.364 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2023-07-20T02:32:54.365 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2023-07-20T02:32:54.365 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2023-07-20T02:32:54.365 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2023-07-20T02:32:54.365 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T02:32:54.365 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 4 2023-07-20T02:32:54.365 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2023-07-20T02:32:54.365 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x5602bc9d5350 2023-07-20T02:32:54.366 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2023-07-20T02:32:54.366 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2023-07-20T02:32:54.366 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2023-07-20T02:32:54.366 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2023-07-20T02:32:54.366 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2023-07-20T02:32:54.367 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2023-07-20T02:32:54.367 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2023-07-20T02:32:54.367 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2023-07-20T02:32:54.367 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2023-07-20T02:32:54.367 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2023-07-20T02:32:54.367 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2023-07-20T02:32:54.367 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2023-07-20T02:32:54.368 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2023-07-20T02:32:54.368 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2023-07-20T02:32:54.368 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2023-07-20T02:32:54.368 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: bloomfilter 2023-07-20T02:32:54.368 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2023-07-20T02:32:54.368 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 5 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: max_auto_readahead_size: 262144 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: prepopulate_block_cache: 0 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: initial_auto_readahead_size: 8192 2023-07-20T02:32:54.369 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T02:32:54.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T02:32:54.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T02:32:54.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression: NoCompression 2023-07-20T02:32:54.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T02:32:54.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T02:32:54.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T02:32:54.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.num_levels: 7 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T02:32:54.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:32:54.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T02:32:54.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T02:32:54.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:32:54.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:32:54.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T02:32:54.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T02:32:54.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T02:32:54.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T02:32:54.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:32:54.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:32:54.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T02:32:54.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T02:32:54.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:32:54.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T02:32:54.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T02:32:54.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T02:32:54.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T02:32:54.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T02:32:54.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T02:32:54.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T02:32:54.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T02:32:54.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T02:32:54.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T02:32:54.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T02:32:54.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T02:32:54.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T02:32:54.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T02:32:54.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T02:32:54.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T02:32:54.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T02:32:54.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T02:32:54.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T02:32:54.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T02:32:54.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T02:32:54.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T02:32:54.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T02:32:54.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T02:32:54.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T02:32:54.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T02:32:54.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T02:32:54.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T02:32:54.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T02:32:54.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T02:32:54.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T02:32:54.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T02:32:54.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.table_properties_collectors: 2023-07-20T02:32:54.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.inplace_update_support: 0 2023-07-20T02:32:54.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T02:32:54.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.bloom_locality: 0 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.max_successive_merges: 0 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T02:32:54.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.ttl: 2592000 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.enable_blob_files: false 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.min_blob_size: 0 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T02:32:54.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T02:32:54.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T02:32:54.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T02:32:54.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T02:32:54.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T02:32:54.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T02:32:54.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T02:32:54.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-20T02:32:54.383 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-20T02:32:54.383 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4faf274c-2ecc-4901-9bdf-d2b5b5de960c 2023-07-20T02:32:54.383 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820374277281, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-20T02:32:54.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-20T02:32:54.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820374278500, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95167, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 93155, "index_size": 294, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13991, "raw_average_key_size": 50, "raw_value_size": 85545, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689820374, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4faf274c-2ecc-4901-9bdf-d2b5b5de960c", "db_session_id": "FBV7HXVTR7SRL1ATPR11", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-20T02:32:54.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820374278575, "job": 1, "event": "recovery_finished"} 2023-07-20T02:32:54.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-20T02:32:54.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T02:32:54.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5602bca6e000 2023-07-20T02:32:54.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: DB pointer 0x5602bca5a000 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:32:54.385 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 94.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:32:54.386 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 94.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:32:54.387 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:32:54.387 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2023-07-20T02:32:54.387 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2023-07-20T02:32:54.387 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:32:54.387 INFO:journalctl@ceph.mon.a.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T02:32:54.387 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:32:54.387 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T02:32:54.388 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T02:32:54.389 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 27.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:32:54.389 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 27.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:32:54.389 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T02:32:54.389 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache BinnedLRUCache@0x5602bc9d5350#7 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T02:32:54.389 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2023-07-20T02:32:54.389 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2023-07-20T02:32:54.389 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???) e1 preinit fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???).mds e1 new map 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???).mds e1 print_map 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T02:32:54.390 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2023-07-20T02:32:54.391 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2023-07-20T02:32:54.391 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2023-07-20T02:32:54.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T02:32:54.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:32:54.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:32:54.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:32:54.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expand map: {default=false} 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta from 'false' to 'false' 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expanded map: {default=false} 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expand map: {default=info} 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta from 'info' to 'info' 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expanded map: {default=info} 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expand map: {default=daemon} 2023-07-20T02:32:54.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expand map: {default=debug} 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: expand_channel_meta expanded map: {default=debug} 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 systemd[1]: Started Ceph mon.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} removed_ranks: {} 2023-07-20T02:32:54.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: fsmap 2023-07-20T02:32:54.394 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T02:32:54.394 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:54 smithi017 ceph-mon[127345]: mgrmap e1: no daemons active 2023-07-20T02:32:55.326 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.818+0000 7ffb19679700 1 Processor -- start 2023-07-20T02:32:55.326 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.818+0000 7ffb19679700 1 -- start start 2023-07-20T02:32:55.326 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.818+0000 7ffb19679700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14104920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:55.326 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.818+0000 7ffb19679700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14104ef0 con 0x7ffb14104500 2023-07-20T02:32:55.326 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.818+0000 7ffb12ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14104920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.818+0000 7ffb12ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14104920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42268/0 (socket says 172.21.15.17:42268) 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.819+0000 7ffb12ffd700 1 -- 172.21.15.17:0/2921635245 learned_addr learned my addr 172.21.15.17:0/2921635245 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.819+0000 7ffb12ffd700 1 -- 172.21.15.17:0/2921635245 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb14107490 con 0x7ffb14104500 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.819+0000 7ffb12ffd700 1 --2- 172.21.15.17:0/2921635245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14104920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb040098a0 tx=0x7ffb04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ea2886e66f2402 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.819+0000 7ffb11ffb700 1 -- 172.21.15.17:0/2921635245 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb04004160 con 0x7ffb14104500 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.819+0000 7ffb11ffb700 1 -- 172.21.15.17:0/2921635245 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb040042c0 con 0x7ffb14104500 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.819+0000 7ffb11ffb700 1 -- 172.21.15.17:0/2921635245 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb04011a00 con 0x7ffb14104500 2023-07-20T02:32:55.327 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 -- 172.21.15.17:0/2921635245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 msgr2=0x7ffb14104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:55.328 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 --2- 172.21.15.17:0/2921635245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14104920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb040098a0 tx=0x7ffb04009bb0 comp rx=0 tx=0).stop 2023-07-20T02:32:55.328 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 -- 172.21.15.17:0/2921635245 shutdown_connections 2023-07-20T02:32:55.328 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 --2- 172.21.15.17:0/2921635245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14104920 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:55.328 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 -- 172.21.15.17:0/2921635245 >> 172.21.15.17:0/2921635245 conn(0x7ffb140ffe10 msgr2=0x7ffb14102270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:55.328 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 -- 172.21.15.17:0/2921635245 shutdown_connections 2023-07-20T02:32:55.328 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 -- 172.21.15.17:0/2921635245 wait complete. 2023-07-20T02:32:55.328 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.820+0000 7ffb19679700 1 Processor -- start 2023-07-20T02:32:55.329 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb19679700 1 -- start start 2023-07-20T02:32:55.329 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb19679700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:55.329 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb19679700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1419a490 con 0x7ffb14104500 2023-07-20T02:32:55.329 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb12ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:55.329 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb12ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42270/0 (socket says 172.21.15.17:42270) 2023-07-20T02:32:55.329 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb12ffd700 1 -- 172.21.15.17:0/2267990410 learned_addr learned my addr 172.21.15.17:0/2267990410 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:55.329 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb12ffd700 1 -- 172.21.15.17:0/2267990410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04009550 con 0x7ffb14104500 2023-07-20T02:32:55.330 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.821+0000 7ffb12ffd700 1 --2- 172.21.15.17:0/2267990410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14199f50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009130 tx=0x7ffb04011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:55.330 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.822+0000 7ffafbfff700 1 -- 172.21.15.17:0/2267990410 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb04003ab0 con 0x7ffb14104500 2023-07-20T02:32:55.330 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.822+0000 7ffafbfff700 1 -- 172.21.15.17:0/2267990410 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb04003c10 con 0x7ffb14104500 2023-07-20T02:32:55.330 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.822+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1419a690 con 0x7ffb14104500 2023-07-20T02:32:55.330 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.822+0000 7ffafbfff700 1 -- 172.21.15.17:0/2267990410 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb040247d0 con 0x7ffb14104500 2023-07-20T02:32:55.330 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.822+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1419aab0 con 0x7ffb14104500 2023-07-20T02:32:55.331 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.822+0000 7ffafbfff700 1 -- 172.21.15.17:0/2267990410 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ffb04024d00 con 0x7ffb14104500 2023-07-20T02:32:55.331 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.822+0000 7ffafbfff700 1 -- 172.21.15.17:0/2267990410 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ffb040193c0 con 0x7ffb14104500 2023-07-20T02:32:55.331 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.823+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb14061da0 con 0x7ffb14104500 2023-07-20T02:32:55.331 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.825+0000 7ffafbfff700 1 -- 172.21.15.17:0/2267990410 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7ffb0401abb0 con 0x7ffb14104500 2023-07-20T02:32:55.331 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.950+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ffb1419dad0 con 0x7ffb14104500 2023-07-20T02:32:55.331 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.953+0000 7ffafbfff700 1 -- 172.21.15.17:0/2267990410 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ffb0401ad50 con 0x7ffb14104500 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.953+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 msgr2=0x7ffb14199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.953+0000 7ffb19679700 1 --2- 172.21.15.17:0/2267990410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14199f50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009130 tx=0x7ffb04011ad0 comp rx=0 tx=0).stop 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.954+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 shutdown_connections 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.954+0000 7ffb19679700 1 --2- 172.21.15.17:0/2267990410 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffb14104500 0x7ffb14199f50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.954+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 >> 172.21.15.17:0/2267990410 conn(0x7ffb140ffe10 msgr2=0x7ffb14101550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.954+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 shutdown_connections 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:54.954+0000 7ffb19679700 1 -- 172.21.15.17:0/2267990410 wait complete. 2023-07-20T02:32:55.332 INFO:teuthology.orchestra.run.smithi017.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-20T02:32:55.383 INFO:teuthology.orchestra.run.smithi017.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:32:55.383 INFO:teuthology.orchestra.run.smithi017.stdout:Creating mgr... 2023-07-20T02:32:55.383 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying port 9283 ... 2023-07-20T02:32:55.384 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying port 8765 ... 2023-07-20T02:32:55.596 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T02:32:55.597 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.x.service: Unit ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.x.service not loaded. 2023-07-20T02:32:55.693 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c.target.wants/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@.service. 2023-07-20T02:32:55.941 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:55 smithi017 systemd[1]: Starting Ceph mgr.x for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:32:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:55 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2267990410' entity='client.admin' 2023-07-20T02:32:56.192 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 podman[127573]: 2023-07-20T02:32:56.302 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:32:56.303 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2023-07-20T02:32:56.310 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:32:56.310 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2023-07-20T02:32:56.310 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2023-07-20T02:32:56.310 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T02:32:56.319 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:32:56.319 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2023-07-20T02:32:56.327 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:32:56.328 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2023-07-20T02:32:56.328 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2023-07-20T02:32:56.328 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-20T02:32:56.328 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr to start... 2023-07-20T02:32:56.328 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr... 2023-07-20T02:32:56.526 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 bash[127573]: 1bf6edd93f658f8899de94e472b4784312d9d93d9c7d153c4d20dcd417a86374 2023-07-20T02:32:56.526 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 systemd[1]: Started Ceph mgr.x for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:32:56.526 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:56.438+0000 7fb32e698200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:32:56.526 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:56.536+0000 7fb32e698200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:32:56.941 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:56.651+0000 7fb32e698200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:32:56.942 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:56.766+0000 7fb32e698200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:32:57.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:32:57 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3701237445' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:32:57.320 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:56.977+0000 7fb32e698200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:32:57.380 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:32:57.381 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:32:57.381 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsid": "9d6085e8-26a5-11ee-9b34-001a4aab830c", 2023-07-20T02:32:57.381 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:32:57.381 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:32:57.381 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:32:57.391 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:32:57.392 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:32:57.392 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:32:57.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:32:57.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:32:57.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:32:57.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:32:57.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:32:57.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:32:57.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:32:57.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:32:57.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:32:57.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:32:57.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:32:57.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:32:50.175992+0000", 2023-07-20T02:32:57.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:32:57.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:32:57.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:32:57.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:32:57.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.871+0000 7f3d27f40700 1 Processor -- start 2023-07-20T02:32:57.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.871+0000 7f3d27f40700 1 -- start start 2023-07-20T02:32:57.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.871+0000 7f3d27f40700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d20104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:57.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.871+0000 7f3d27f40700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20073f30 con 0x7f3d20104ae0 2023-07-20T02:32:57.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.872+0000 7f3d25cdc700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d20104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:57.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.872+0000 7f3d25cdc700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d20104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42276/0 (socket says 172.21.15.17:42276) 2023-07-20T02:32:57.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.872+0000 7f3d25cdc700 1 -- 172.21.15.17:0/464869058 learned_addr learned my addr 172.21.15.17:0/464869058 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:57.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.872+0000 7f3d25cdc700 1 -- 172.21.15.17:0/464869058 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20105440 con 0x7f3d20104ae0 2023-07-20T02:32:57.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.872+0000 7f3d25cdc700 1 --2- 172.21.15.17:0/464869058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d20104f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0098a0 tx=0x7f3d1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f8480b5ca4760d server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:57.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.872+0000 7f3d24cda700 1 -- 172.21.15.17:0/464869058 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d1c004160 con 0x7f3d20104ae0 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d24cda700 1 -- 172.21.15.17:0/464869058 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d1c0042c0 con 0x7f3d20104ae0 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d24cda700 1 -- 172.21.15.17:0/464869058 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d1c011a00 con 0x7f3d20104ae0 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d27f40700 1 -- 172.21.15.17:0/464869058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 msgr2=0x7f3d20104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d27f40700 1 --2- 172.21.15.17:0/464869058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d20104f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0098a0 tx=0x7f3d1c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d27f40700 1 -- 172.21.15.17:0/464869058 shutdown_connections 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d27f40700 1 --2- 172.21.15.17:0/464869058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d20104f00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d27f40700 1 -- 172.21.15.17:0/464869058 >> 172.21.15.17:0/464869058 conn(0x7f3d20100680 msgr2=0x7f3d20102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:57.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d27f40700 1 -- 172.21.15.17:0/464869058 shutdown_connections 2023-07-20T02:32:57.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.873+0000 7f3d27f40700 1 -- 172.21.15.17:0/464869058 wait complete. 2023-07-20T02:32:57.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.874+0000 7f3d27f40700 1 Processor -- start 2023-07-20T02:32:57.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.874+0000 7f3d27f40700 1 -- start start 2023-07-20T02:32:57.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.874+0000 7f3d27f40700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d2019c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:32:57.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.874+0000 7f3d27f40700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2019c660 con 0x7f3d20104ae0 2023-07-20T02:32:57.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.874+0000 7f3d25cdc700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d2019c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:32:57.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.874+0000 7f3d25cdc700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d2019c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42278/0 (socket says 172.21.15.17:42278) 2023-07-20T02:32:57.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.874+0000 7f3d25cdc700 1 -- 172.21.15.17:0/3701237445 learned_addr learned my addr 172.21.15.17:0/3701237445 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:32:57.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d25cdc700 1 -- 172.21.15.17:0/3701237445 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c009550 con 0x7f3d20104ae0 2023-07-20T02:32:57.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d25cdc700 1 --2- 172.21.15.17:0/3701237445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d2019c120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009fb0 tx=0x7f3d1c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:32:57.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d16ffd700 1 -- 172.21.15.17:0/3701237445 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d1c003a40 con 0x7f3d20104ae0 2023-07-20T02:32:57.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2019c860 con 0x7f3d20104ae0 2023-07-20T02:32:57.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d16ffd700 1 -- 172.21.15.17:0/3701237445 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d1c003ba0 con 0x7f3d20104ae0 2023-07-20T02:32:57.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d16ffd700 1 -- 172.21.15.17:0/3701237445 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d1c01a750 con 0x7f3d20104ae0 2023-07-20T02:32:57.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2019cc80 con 0x7f3d20104ae0 2023-07-20T02:32:57.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d16ffd700 1 -- 172.21.15.17:0/3701237445 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3d1c01ac80 con 0x7f3d20104ae0 2023-07-20T02:32:57.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.875+0000 7f3d16ffd700 1 -- 172.21.15.17:0/3701237445 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3d1c0193c0 con 0x7f3d20104ae0 2023-07-20T02:32:57.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.876+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d20195a20 con 0x7f3d20104ae0 2023-07-20T02:32:57.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:56.878+0000 7f3d16ffd700 1 -- 172.21.15.17:0/3701237445 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f3d1c02de20 con 0x7f3d20104ae0 2023-07-20T02:32:57.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.011+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3d2002c970 con 0x7f3d20104ae0 2023-07-20T02:32:57.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.012+0000 7f3d16ffd700 1 -- 172.21.15.17:0/3701237445 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f3d1c024b60 con 0x7f3d20104ae0 2023-07-20T02:32:57.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.013+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 msgr2=0x7f3d2019c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:32:57.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.013+0000 7f3d27f40700 1 --2- 172.21.15.17:0/3701237445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d2019c120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009fb0 tx=0x7f3d1c011ad0 comp rx=0 tx=0).stop 2023-07-20T02:32:57.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.013+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 shutdown_connections 2023-07-20T02:32:57.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.013+0000 7f3d27f40700 1 --2- 172.21.15.17:0/3701237445 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3d20104ae0 0x7f3d2019c120 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:32:57.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.013+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 >> 172.21.15.17:0/3701237445 conn(0x7f3d20100680 msgr2=0x7f3d201906a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:32:57.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.013+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 shutdown_connections 2023-07-20T02:32:57.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:57.013+0000 7f3d27f40700 1 -- 172.21.15.17:0/3701237445 wait complete. 2023-07-20T02:32:57.407 INFO:teuthology.orchestra.run.smithi017.stdout:mgr not available, waiting (1/15)... 2023-07-20T02:32:57.639 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:57.382+0000 7fb32e698200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:32:57.639 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:57.493+0000 7fb32e698200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:32:57.942 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:57.650+0000 7fb32e698200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:32:57.942 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:57.766+0000 7fb32e698200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:32:58.691 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:58 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:58.277+0000 7fb32e698200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:32:59.442 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:58 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:58.974+0000 7fb32e698200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:32:59.442 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:32:59 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:32:59.164+0000 7fb32e698200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:33:00.339 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:33:00.339 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:33:00.340 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsid": "9d6085e8-26a5-11ee-9b34-001a4aab830c", 2023-07-20T02:33:00.340 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:33:00.340 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:33:00.340 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:33:00.340 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:33:00.340 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:00.341 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:33:00.341 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:33:00.341 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:33:00.341 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:00.341 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:33:00.342 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:33:00.342 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:00.342 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-20T02:33:00.342 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:33:00.342 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:00.342 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:33:00.343 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:33:00.344 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:33:00.344 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:33:00.344 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:00.344 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:33:00.344 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:33:00.344 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:33:00.344 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:33:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:00.346 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:33:00.346 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:33:00.346 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:00.346 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:33:00.346 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T02:33:00.346 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:33:00.346 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:33:00.347 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:00.348 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:32:50.175992+0000", 2023-07-20T02:33:00.348 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:33:00.348 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:00.348 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:33:00.348 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:33:00.348 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.890+0000 7f13795a3700 1 Processor -- start 2023-07-20T02:33:00.348 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.890+0000 7f13795a3700 1 -- start start 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.890+0000 7f13795a3700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f1374104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.890+0000 7f13795a3700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1374073f30 con 0x7f1374104ae0 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1372ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f1374104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1372ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f1374104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42280/0 (socket says 172.21.15.17:42280) 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1372ffd700 1 -- 172.21.15.17:0/3452338620 learned_addr learned my addr 172.21.15.17:0/3452338620 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1372ffd700 1 -- 172.21.15.17:0/3452338620 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1374105440 con 0x7f1374104ae0 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1372ffd700 1 --2- 172.21.15.17:0/3452338620 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f1374104f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f13640098a0 tx=0x7f1364009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172cd14d220f40ae server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:00.349 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1371ffb700 1 -- 172.21.15.17:0/3452338620 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1364004160 con 0x7f1374104ae0 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1371ffb700 1 -- 172.21.15.17:0/3452338620 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13640042c0 con 0x7f1374104ae0 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.891+0000 7f1371ffb700 1 -- 172.21.15.17:0/3452338620 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1364011a00 con 0x7f1374104ae0 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.892+0000 7f13795a3700 1 -- 172.21.15.17:0/3452338620 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 msgr2=0x7f1374104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.892+0000 7f13795a3700 1 --2- 172.21.15.17:0/3452338620 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f1374104f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f13640098a0 tx=0x7f1364009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.892+0000 7f13795a3700 1 -- 172.21.15.17:0/3452338620 shutdown_connections 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.892+0000 7f13795a3700 1 --2- 172.21.15.17:0/3452338620 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f1374104f00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.892+0000 7f13795a3700 1 -- 172.21.15.17:0/3452338620 >> 172.21.15.17:0/3452338620 conn(0x7f1374100680 msgr2=0x7f1374102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:00.350 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.892+0000 7f13795a3700 1 -- 172.21.15.17:0/3452338620 shutdown_connections 2023-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.892+0000 7f13795a3700 1 -- 172.21.15.17:0/3452338620 wait complete. 2023-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.893+0000 7f13795a3700 1 Processor -- start 2023-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.893+0000 7f13795a3700 1 -- start start 2023-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.893+0000 7f13795a3700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f137419c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.893+0000 7f13795a3700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137419c660 con 0x7f1374104ae0 2023-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.893+0000 7f1372ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f137419c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.893+0000 7f1372ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f137419c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42282/0 (socket says 172.21.15.17:42282) 2023-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.893+0000 7f1372ffd700 1 -- 172.21.15.17:0/2852504791 learned_addr learned my addr 172.21.15.17:0/2852504791 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f1372ffd700 1 -- 172.21.15.17:0/2852504791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1364009550 con 0x7f1374104ae0 2023-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f1372ffd700 1 --2- 172.21.15.17:0/2852504791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f137419c120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1364009fb0 tx=0x7f1364011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f135bfff700 1 -- 172.21.15.17:0/2852504791 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1364003a40 con 0x7f1374104ae0 2023-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f135bfff700 1 -- 172.21.15.17:0/2852504791 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1364003ba0 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137419c860 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f135bfff700 1 -- 172.21.15.17:0/2852504791 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f136401a750 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137419cc80 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f135bfff700 1 -- 172.21.15.17:0/2852504791 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f136401acd0 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.894+0000 7f135bfff700 1 -- 172.21.15.17:0/2852504791 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f13640193c0 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.896+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1374195a20 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:32:59.898+0000 7f135bfff700 1 -- 172.21.15.17:0/2852504791 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f136402de40 con 0x7f1374104ae0 2023-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.031+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f137402cb90 con 0x7f1374104ae0 2023-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.031+0000 7f135bfff700 1 -- 172.21.15.17:0/2852504791 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f1364024b60 con 0x7f1374104ae0 2023-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.032+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 msgr2=0x7f137419c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.033+0000 7f13795a3700 1 --2- 172.21.15.17:0/2852504791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f137419c120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1364009fb0 tx=0x7f1364011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.033+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 shutdown_connections 2023-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.033+0000 7f13795a3700 1 --2- 172.21.15.17:0/2852504791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1374104ae0 0x7f137419c120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.033+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 >> 172.21.15.17:0/2852504791 conn(0x7f1374100680 msgr2=0x7f13741906a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.033+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 shutdown_connections 2023-07-20T02:33:00.355 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:00.033+0000 7f13795a3700 1 -- 172.21.15.17:0/2852504791 wait complete. 2023-07-20T02:33:00.355 INFO:teuthology.orchestra.run.smithi017.stdout:mgr not available, waiting (2/15)... 2023-07-20T02:33:00.416 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:00 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2852504791' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:33:00.417 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:00.157+0000 7fb32e698200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:33:00.417 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:00.268+0000 7fb32e698200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:33:00.691 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:00.488+0000 7fb32e698200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:33:01.084 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:00.837+0000 7fb32e698200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:33:01.442 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:01.095+0000 7fb32e698200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:33:01.442 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:01.277+0000 7fb32e698200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:33:01.942 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:01.603+0000 7fb32e698200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:33:02.692 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:02.433+0000 7fb32e698200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:33:02.692 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:02.597+0000 7fb32e698200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:33:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:02 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4210234329' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:33:03.192 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:02.844+0000 7fb32e698200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:33:03.193 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:02.982+0000 7fb32e698200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:33:03.692 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:03 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:03.258+0000 7fb32e698200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:33:04.692 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:04.303+0000 7fb32e698200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:33:05.389 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:33:05.389 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:33:05.390 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsid": "9d6085e8-26a5-11ee-9b34-001a4aab830c", 2023-07-20T02:33:05.390 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:33:05.390 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:33:05.390 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:33:05.391 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:33:05.391 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:05.391 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:33:05.391 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:33:05.392 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:33:05.392 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:05.392 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:33:05.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:33:05.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:05.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-20T02:33:05.393 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:33:05.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:05.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:33:05.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:33:05.394 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:05.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:33:05.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:05.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:33:05.395 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:33:05.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:33:05.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:33:05.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:33:05.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:33:05.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:05.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:33:05.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:33:05.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:05.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:33:05.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:05.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:33:05.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:33:05.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:05.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:33:05.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T02:33:05.406 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:33:05.408 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:05.408 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:32:50.175992+0000", 2023-07-20T02:33:05.408 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:33:05.408 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:05.408 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:33:05.408 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:33:05.408 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.803+0000 7f3474308700 1 Processor -- start 2023-07-20T02:33:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.803+0000 7f3474308700 1 -- start start 2023-07-20T02:33:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.803+0000 7f3474308700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c079180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.803+0000 7f3474308700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c079750 con 0x7f346c07ad20 2023-07-20T02:33:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34720a4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c079180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34720a4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c079180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42284/0 (socket says 172.21.15.17:42284) 2023-07-20T02:33:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34720a4700 1 -- 172.21.15.17:0/2091615006 learned_addr learned my addr 172.21.15.17:0/2091615006 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:05.409 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34720a4700 1 -- 172.21.15.17:0/2091615006 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c079890 con 0x7f346c07ad20 2023-07-20T02:33:05.410 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34720a4700 1 --2- 172.21.15.17:0/2091615006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c079180 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f34680098a0 tx=0x7f3468009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e38bd85e564140a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:05.410 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34710a2700 1 -- 172.21.15.17:0/2091615006 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3468004160 con 0x7f346c07ad20 2023-07-20T02:33:05.410 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34710a2700 1 -- 172.21.15.17:0/2091615006 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34680042c0 con 0x7f346c07ad20 2023-07-20T02:33:05.410 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.804+0000 7f34710a2700 1 -- 172.21.15.17:0/2091615006 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3468011a00 con 0x7f346c07ad20 2023-07-20T02:33:05.411 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 -- 172.21.15.17:0/2091615006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 msgr2=0x7f346c079180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:05.411 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 --2- 172.21.15.17:0/2091615006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c079180 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f34680098a0 tx=0x7f3468009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:05.411 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 -- 172.21.15.17:0/2091615006 shutdown_connections 2023-07-20T02:33:05.411 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 --2- 172.21.15.17:0/2091615006 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c079180 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:05.412 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 -- 172.21.15.17:0/2091615006 >> 172.21.15.17:0/2091615006 conn(0x7f346c101770 msgr2=0x7f346c103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:05.412 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 -- 172.21.15.17:0/2091615006 shutdown_connections 2023-07-20T02:33:05.412 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 -- 172.21.15.17:0/2091615006 wait complete. 2023-07-20T02:33:05.412 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.805+0000 7f3474308700 1 Processor -- start 2023-07-20T02:33:05.413 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f3474308700 1 -- start start 2023-07-20T02:33:05.413 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f3474308700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c19c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:05.413 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f3474308700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c19c6a0 con 0x7f346c07ad20 2023-07-20T02:33:05.413 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f34720a4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c19c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:05.414 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f34720a4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c19c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42286/0 (socket says 172.21.15.17:42286) 2023-07-20T02:33:05.414 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f34720a4700 1 -- 172.21.15.17:0/4210234329 learned_addr learned my addr 172.21.15.17:0/4210234329 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:05.414 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f34720a4700 1 -- 172.21.15.17:0/4210234329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468009550 con 0x7f346c07ad20 2023-07-20T02:33:05.414 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f34720a4700 1 --2- 172.21.15.17:0/4210234329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c19c160 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3468009130 tx=0x7f3468011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:05.415 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f34637fe700 1 -- 172.21.15.17:0/4210234329 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3468003ab0 con 0x7f346c07ad20 2023-07-20T02:33:05.415 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.806+0000 7f34637fe700 1 -- 172.21.15.17:0/4210234329 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3468003c10 con 0x7f346c07ad20 2023-07-20T02:33:05.415 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.807+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c19c8a0 con 0x7f346c07ad20 2023-07-20T02:33:05.416 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.807+0000 7f34637fe700 1 -- 172.21.15.17:0/4210234329 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34680247d0 con 0x7f346c07ad20 2023-07-20T02:33:05.416 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.807+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c19ccc0 con 0x7f346c07ad20 2023-07-20T02:33:05.416 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.807+0000 7f34637fe700 1 -- 172.21.15.17:0/4210234329 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3468024d00 con 0x7f346c07ad20 2023-07-20T02:33:05.417 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.807+0000 7f34637fe700 1 -- 172.21.15.17:0/4210234329 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34680193c0 con 0x7f346c07ad20 2023-07-20T02:33:05.417 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.808+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346c1961d0 con 0x7f346c07ad20 2023-07-20T02:33:05.417 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.810+0000 7f34637fe700 1 -- 172.21.15.17:0/4210234329 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f346801abb0 con 0x7f346c07ad20 2023-07-20T02:33:05.417 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.943+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f346c061da0 con 0x7f346c07ad20 2023-07-20T02:33:05.418 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.943+0000 7f34637fe700 1 -- 172.21.15.17:0/4210234329 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f3468035090 con 0x7f346c07ad20 2023-07-20T02:33:05.418 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.944+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 msgr2=0x7f346c19c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:05.418 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.944+0000 7f3474308700 1 --2- 172.21.15.17:0/4210234329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c19c160 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3468009130 tx=0x7f3468011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:05.419 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.944+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 shutdown_connections 2023-07-20T02:33:05.419 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.944+0000 7f3474308700 1 --2- 172.21.15.17:0/4210234329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f346c07ad20 0x7f346c19c160 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:05.419 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.944+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 >> 172.21.15.17:0/4210234329 conn(0x7f346c101770 msgr2=0x7f346c102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:05.419 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.945+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 shutdown_connections 2023-07-20T02:33:05.419 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:02.945+0000 7f3474308700 1 -- 172.21.15.17:0/4210234329 wait complete. 2023-07-20T02:33:05.420 INFO:teuthology.orchestra.run.smithi017.stdout:mgr not available, waiting (3/15)... 2023-07-20T02:33:05.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: Activating manager daemon x 2023-07-20T02:33:05.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: mgrmap e2: x(active, starting, since 0.00286743s) 2023-07-20T02:33:05.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T02:33:05.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T02:33:05.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T02:33:05.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:33:05.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T02:33:05.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: Manager daemon x is now available 2023-07-20T02:33:05.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T02:33:05.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T02:33:05.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' 2023-07-20T02:33:05.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' 2023-07-20T02:33:05.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:05 smithi017 ceph-mon[127345]: from='mgr.14100 172.21.15.17:0/1020872790' entity='mgr.x' 2023-07-20T02:33:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:06 smithi017 ceph-mon[127345]: mgrmap e3: x(active, since 1.00708s) 2023-07-20T02:33:08.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:08 smithi017 ceph-mon[127345]: mgrmap e4: x(active, since 3s) 2023-07-20T02:33:08.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:33:08.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:33:08.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsid": "9d6085e8-26a5-11ee-9b34-001a4aab830c", 2023-07-20T02:33:08.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:33:08.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:33:08.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:33:08.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:33:08.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:08.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:33:08.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:33:08.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:33:08.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:08.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:33:08.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:33:08.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:08.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2023-07-20T02:33:08.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:33:08.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:08.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:33:08.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:33:08.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:08.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:33:08.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:08.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:33:08.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:33:08.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:33:08.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:33:08.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:33:08.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:33:08.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:33:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:33:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:33:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:33:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:33:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:33:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:33:08.758 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:33:08.758 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:33:08.758 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:08.758 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:33:08.758 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:08.758 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:33:08.759 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:33:08.759 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:08.759 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:33:08.759 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T02:33:08.759 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:33:08.759 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:33:08.759 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:33:08.760 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:33:08.760 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:33:08.760 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:33:08.760 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:33:08.760 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:08.760 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:33:08.761 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:33:08.761 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:32:50.175992+0000", 2023-07-20T02:33:08.761 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:33:08.761 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:33:08.761 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:33:08.761 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:33:08.761 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.926+0000 7f06fc7f8700 1 Processor -- start 2023-07-20T02:33:08.762 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.926+0000 7f06fc7f8700 1 -- start start 2023-07-20T02:33:08.762 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.926+0000 7f06fc7f8700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f4104ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:08.762 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.926+0000 7f06fc7f8700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4079dc0 con 0x7f06f4104ad0 2023-07-20T02:33:08.762 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.926+0000 7f06fa594700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f4104ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:08.762 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.926+0000 7f06fa594700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f4104ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42312/0 (socket says 172.21.15.17:42312) 2023-07-20T02:33:08.762 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.926+0000 7f06fa594700 1 -- 172.21.15.17:0/3275206326 learned_addr learned my addr 172.21.15.17:0/3275206326 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:08.763 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.927+0000 7f06fa594700 1 -- 172.21.15.17:0/3275206326 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f4105430 con 0x7f06f4104ad0 2023-07-20T02:33:08.763 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.927+0000 7f06fa594700 1 --2- 172.21.15.17:0/3275206326 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f4104ef0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f06f00098a0 tx=0x7f06f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e761db51f6a794 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:08.763 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.927+0000 7f06f9592700 1 -- 172.21.15.17:0/3275206326 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f0004160 con 0x7f06f4104ad0 2023-07-20T02:33:08.763 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.927+0000 7f06f9592700 1 -- 172.21.15.17:0/3275206326 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06f00042c0 con 0x7f06f4104ad0 2023-07-20T02:33:08.763 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.927+0000 7f06f9592700 1 -- 172.21.15.17:0/3275206326 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f0011a00 con 0x7f06f4104ad0 2023-07-20T02:33:08.763 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 -- 172.21.15.17:0/3275206326 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 msgr2=0x7f06f4104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:08.763 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 --2- 172.21.15.17:0/3275206326 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f4104ef0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f06f00098a0 tx=0x7f06f0009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:08.764 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 -- 172.21.15.17:0/3275206326 shutdown_connections 2023-07-20T02:33:08.764 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 --2- 172.21.15.17:0/3275206326 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f4104ef0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:08.764 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 -- 172.21.15.17:0/3275206326 >> 172.21.15.17:0/3275206326 conn(0x7f06f41006b0 msgr2=0x7f06f4102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:08.764 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 -- 172.21.15.17:0/3275206326 shutdown_connections 2023-07-20T02:33:08.764 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 -- 172.21.15.17:0/3275206326 wait complete. 2023-07-20T02:33:08.764 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 Processor -- start 2023-07-20T02:33:08.765 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.928+0000 7f06fc7f8700 1 -- start start 2023-07-20T02:33:08.765 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fc7f8700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f419c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:08.765 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fc7f8700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f419c670 con 0x7f06f4104ad0 2023-07-20T02:33:08.765 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fa594700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f419c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:08.765 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fa594700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f419c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42314/0 (socket says 172.21.15.17:42314) 2023-07-20T02:33:08.765 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fa594700 1 -- 172.21.15.17:0/4081899320 learned_addr learned my addr 172.21.15.17:0/4081899320 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:08.765 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fa594700 1 -- 172.21.15.17:0/4081899320 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0009550 con 0x7f06f4104ad0 2023-07-20T02:33:08.766 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fa594700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f419c130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009130 tx=0x7f06f0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:08.766 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06eb7fe700 1 -- 172.21.15.17:0/4081899320 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f0003b70 con 0x7f06f4104ad0 2023-07-20T02:33:08.766 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f419c870 con 0x7f06f4104ad0 2023-07-20T02:33:08.766 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06eb7fe700 1 -- 172.21.15.17:0/4081899320 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06f0003cd0 con 0x7f06f4104ad0 2023-07-20T02:33:08.766 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06eb7fe700 1 -- 172.21.15.17:0/4081899320 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f00197d0 con 0x7f06f4104ad0 2023-07-20T02:33:08.766 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.929+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f419cc90 con 0x7f06f4104ad0 2023-07-20T02:33:08.766 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.930+0000 7f06eb7fe700 1 -- 172.21.15.17:0/4081899320 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f06f001abb0 con 0x7f06f4104ad0 2023-07-20T02:33:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.930+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f4195a10 con 0x7f06f4104ad0 2023-07-20T02:33:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.931+0000 7f06eb7fe700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f06e00386c0 0x7f06e003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.931+0000 7f06eb7fe700 1 -- 172.21.15.17:0/4081899320 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f06f004c890 con 0x7f06f4104ad0 2023-07-20T02:33:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.931+0000 7f06f9d93700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f06e00386c0 0x7f06e003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.931+0000 7f06f9d93700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f06e00386c0 0x7f06e003ab80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006fd0 tx=0x7f06e4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:07.936+0000 7f06eb7fe700 1 -- 172.21.15.17:0/4081899320 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f06f001a430 con 0x7f06f4104ad0 2023-07-20T02:33:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.448+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f06f4061da0 con 0x7f06f4104ad0 2023-07-20T02:33:08.768 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.448+0000 7f06eb7fe700 1 -- 172.21.15.17:0/4081899320 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f06f0023370 con 0x7f06f4104ad0 2023-07-20T02:33:08.768 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f06e00386c0 msgr2=0x7f06e003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:08.768 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f06e00386c0 0x7f06e003ab80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006fd0 tx=0x7f06e4006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:08.768 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 msgr2=0x7f06f419c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:08.768 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f419c130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009130 tx=0x7f06f0011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:08.768 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 shutdown_connections 2023-07-20T02:33:08.768 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f06e00386c0 0x7f06e003ab80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:08.769 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 --2- 172.21.15.17:0/4081899320 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f06f4104ad0 0x7f06f419c130 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:08.769 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 >> 172.21.15.17:0/4081899320 conn(0x7f06f41006b0 msgr2=0x7f06f4190690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:08.769 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 shutdown_connections 2023-07-20T02:33:08.769 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:08.451+0000 7f06fc7f8700 1 -- 172.21.15.17:0/4081899320 wait complete. 2023-07-20T02:33:08.769 INFO:teuthology.orchestra.run.smithi017.stdout:mgr is available 2023-07-20T02:33:09.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:09 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4081899320' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:33:09.994 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:33:09.995 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T02:33:09.995 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout fsid = 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:33:09.995 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T02:33:09.995 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T02:33:09.995 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T02:33:09.995 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T02:33:09.996 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:33:09.996 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T02:33:09.996 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T02:33:09.996 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2023-07-20T02:33:09.997 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T02:33:09.997 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T02:33:09.997 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T02:33:09.997 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T02:33:09.998 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T02:33:09.998 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T02:33:09.998 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.248+0000 7f8e4ef48700 1 Processor -- start 2023-07-20T02:33:09.998 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.248+0000 7f8e4ef48700 1 -- start start 2023-07-20T02:33:09.999 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.248+0000 7f8e4ef48700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48109090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:09.999 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.248+0000 7f8e4ef48700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48079e00 con 0x7f8e48106ca0 2023-07-20T02:33:09.999 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.248+0000 7f8e4cce4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:09.999 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.248+0000 7f8e4cce4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48109090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42318/0 (socket says 172.21.15.17:42318) 2023-07-20T02:33:10.000 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.248+0000 7f8e4cce4700 1 -- 172.21.15.17:0/1463895262 learned_addr learned my addr 172.21.15.17:0/1463895262 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:10.000 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.249+0000 7f8e4cce4700 1 -- 172.21.15.17:0/1463895262 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e481095d0 con 0x7f8e48106ca0 2023-07-20T02:33:10.000 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.249+0000 7f8e4cce4700 1 --2- 172.21.15.17:0/1463895262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48109090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0098a0 tx=0x7f8e3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48cf7b9db83ca54f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:10.000 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.249+0000 7f8e477fe700 1 -- 172.21.15.17:0/1463895262 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e3c004160 con 0x7f8e48106ca0 2023-07-20T02:33:10.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.249+0000 7f8e477fe700 1 -- 172.21.15.17:0/1463895262 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e3c0042c0 con 0x7f8e48106ca0 2023-07-20T02:33:10.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.249+0000 7f8e477fe700 1 -- 172.21.15.17:0/1463895262 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e3c011a00 con 0x7f8e48106ca0 2023-07-20T02:33:10.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.249+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1463895262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 msgr2=0x7f8e48109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:10.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.249+0000 7f8e4ef48700 1 --2- 172.21.15.17:0/1463895262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48109090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0098a0 tx=0x7f8e3c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:10.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.250+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1463895262 shutdown_connections 2023-07-20T02:33:10.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.250+0000 7f8e4ef48700 1 --2- 172.21.15.17:0/1463895262 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48109090 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:10.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.250+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1463895262 >> 172.21.15.17:0/1463895262 conn(0x7f8e481006b0 msgr2=0x7f8e48102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:10.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.250+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1463895262 shutdown_connections 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.250+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1463895262 wait complete. 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.250+0000 7f8e4ef48700 1 Processor -- start 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.250+0000 7f8e4ef48700 1 -- start start 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4ef48700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4ef48700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4819a470 con 0x7f8e48106ca0 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4cce4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4cce4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42320/0 (socket says 172.21.15.17:42320) 2023-07-20T02:33:10.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4cce4700 1 -- 172.21.15.17:0/1700021164 learned_addr learned my addr 172.21.15.17:0/1700021164 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:10.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4cce4700 1 -- 172.21.15.17:0/1700021164 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c009550 con 0x7f8e48106ca0 2023-07-20T02:33:10.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4cce4700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48199f30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009130 tx=0x7f8e3c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:10.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e45ffb700 1 -- 172.21.15.17:0/1700021164 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e3c003b70 con 0x7f8e48106ca0 2023-07-20T02:33:10.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4819a670 con 0x7f8e48106ca0 2023-07-20T02:33:10.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e45ffb700 1 -- 172.21.15.17:0/1700021164 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e3c003cd0 con 0x7f8e48106ca0 2023-07-20T02:33:10.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.251+0000 7f8e45ffb700 1 -- 172.21.15.17:0/1700021164 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e3c0197d0 con 0x7f8e48106ca0 2023-07-20T02:33:10.004 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.252+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4819aa90 con 0x7f8e48106ca0 2023-07-20T02:33:10.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.252+0000 7f8e45ffb700 1 -- 172.21.15.17:0/1700021164 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f8e3c019a10 con 0x7f8e48106ca0 2023-07-20T02:33:10.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.253+0000 7f8e45ffb700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8e380386c0 0x7f8e3803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:10.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.253+0000 7f8e45ffb700 1 -- 172.21.15.17:0/1700021164 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8e3c04caf0 con 0x7f8e48106ca0 2023-07-20T02:33:10.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.253+0000 7f8e47fff700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8e380386c0 0x7f8e3803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:10.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.253+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e48193fe0 con 0x7f8e48106ca0 2023-07-20T02:33:10.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.253+0000 7f8e47fff700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8e380386c0 0x7f8e3803ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e34006fd0 tx=0x7f8e34006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:10.005 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.258+0000 7f8e45ffb700 1 -- 172.21.15.17:0/1700021164 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8e3c018db0 con 0x7f8e48106ca0 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.614+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8e4802cbf0 con 0x7f8e48106ca0 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.615+0000 7f8e45ffb700 1 -- 172.21.15.17:0/1700021164 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f8e3c018350 con 0x7f8e48106ca0 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.617+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8e380386c0 msgr2=0x7f8e3803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.617+0000 7f8e4ef48700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8e380386c0 0x7f8e3803ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e34006fd0 tx=0x7f8e34006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.617+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 msgr2=0x7f8e48199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.617+0000 7f8e4ef48700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48199f30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009130 tx=0x7f8e3c011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.618+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 shutdown_connections 2023-07-20T02:33:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.618+0000 7f8e4ef48700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8e380386c0 0x7f8e3803ab80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:10.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.618+0000 7f8e4ef48700 1 --2- 172.21.15.17:0/1700021164 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8e48106ca0 0x7f8e48199f30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:10.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.618+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 >> 172.21.15.17:0/1700021164 conn(0x7f8e481006b0 msgr2=0x7f8e48101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:10.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.618+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 shutdown_connections 2023-07-20T02:33:10.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:09.618+0000 7f8e4ef48700 1 -- 172.21.15.17:0/1700021164 wait complete. 2023-07-20T02:33:10.008 INFO:teuthology.orchestra.run.smithi017.stdout:Enabling cephadm module... 2023-07-20T02:33:10.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:10 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1700021164' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T02:33:11.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:11 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/595143586' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-20T02:33:11.603 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: ignoring --setuser ceph since I am not root 2023-07-20T02:33:11.603 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: ignoring --setgroup ceph since I am not root 2023-07-20T02:33:11.603 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:11.516+0000 7f3385e41200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:33:11.942 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:11.613+0000 7f3385e41200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:33:11.942 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:11.725+0000 7f3385e41200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:33:11.943 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:11.836+0000 7f3385e41200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:33:12.315 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:12.044+0000 7f3385e41200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:33:12.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:12 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/595143586' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-20T02:33:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:12 smithi017 ceph-mon[127345]: mgrmap e5: x(active, since 7s) 2023-07-20T02:33:12.740 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:12.440+0000 7f3385e41200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:33:12.740 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:12.549+0000 7f3385e41200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:33:12.740 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:12.700+0000 7f3385e41200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:33:12.942 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:12.815+0000 7f3385e41200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:33:13.692 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:13.326+0000 7f3385e41200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:33:13.744 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.469+0000 7f8dd4179700 1 Processor -- start 2023-07-20T02:33:13.745 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.469+0000 7f8dd4179700 1 -- start start 2023-07-20T02:33:13.745 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.469+0000 7f8dd4179700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc104910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:13.745 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.469+0000 7f8dd4179700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc104ee0 con 0x7f8dcc1044f0 2023-07-20T02:33:13.745 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd1f15700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc104910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:13.745 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd1f15700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc104910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42324/0 (socket says 172.21.15.17:42324) 2023-07-20T02:33:13.745 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd1f15700 1 -- 172.21.15.17:0/2271106172 learned_addr learned my addr 172.21.15.17:0/2271106172 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:13.745 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd1f15700 1 -- 172.21.15.17:0/2271106172 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc105020 con 0x7f8dcc1044f0 2023-07-20T02:33:13.746 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd1f15700 1 --2- 172.21.15.17:0/2271106172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc104910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0098a0 tx=0x7f8dbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e2fa63b1c0a2a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:13.746 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd0f13700 1 -- 172.21.15.17:0/2271106172 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8dbc004160 con 0x7f8dcc1044f0 2023-07-20T02:33:13.746 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd0f13700 1 -- 172.21.15.17:0/2271106172 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dbc0042c0 con 0x7f8dcc1044f0 2023-07-20T02:33:13.746 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.470+0000 7f8dd0f13700 1 -- 172.21.15.17:0/2271106172 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8dbc011a00 con 0x7f8dcc1044f0 2023-07-20T02:33:13.746 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.471+0000 7f8dd4179700 1 -- 172.21.15.17:0/2271106172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 msgr2=0x7f8dcc104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:13.746 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.471+0000 7f8dd4179700 1 --2- 172.21.15.17:0/2271106172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc104910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0098a0 tx=0x7f8dbc009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.471+0000 7f8dd4179700 1 -- 172.21.15.17:0/2271106172 shutdown_connections 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.471+0000 7f8dd4179700 1 --2- 172.21.15.17:0/2271106172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc104910 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.471+0000 7f8dd4179700 1 -- 172.21.15.17:0/2271106172 >> 172.21.15.17:0/2271106172 conn(0x7f8dcc0ffe10 msgr2=0x7f8dcc102270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.471+0000 7f8dd4179700 1 -- 172.21.15.17:0/2271106172 shutdown_connections 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.471+0000 7f8dd4179700 1 -- 172.21.15.17:0/2271106172 wait complete. 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd4179700 1 Processor -- start 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd4179700 1 -- start start 2023-07-20T02:33:13.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd4179700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd4179700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc19a430 con 0x7f8dcc1044f0 2023-07-20T02:33:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd1f15700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd1f15700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42326/0 (socket says 172.21.15.17:42326) 2023-07-20T02:33:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd1f15700 1 -- 172.21.15.17:0/595143586 learned_addr learned my addr 172.21.15.17:0/595143586 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd1f15700 1 -- 172.21.15.17:0/595143586 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc009550 con 0x7f8dcc1044f0 2023-07-20T02:33:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.472+0000 7f8dd1f15700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc199ef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009fb0 tx=0x7f8dbc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.473+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8dbc003a40 con 0x7f8dcc1044f0 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.473+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dcc19a690 con 0x7f8dcc1044f0 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.473+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dbc003ba0 con 0x7f8dcc1044f0 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.473+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8dbc01a750 con 0x7f8dcc1044f0 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.473+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dcc19ab10 con 0x7f8dcc1044f0 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.474+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f8dbc01a9d0 con 0x7f8dcc1044f0 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.474+0000 7f8dc2ffd700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8db8038670 0x7f8db803ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.474+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dcc061da0 con 0x7f8dcc1044f0 2023-07-20T02:33:13.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.474+0000 7f8dd1714700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8db8038670 0x7f8db803ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:13.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.474+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8dbc04c430 con 0x7f8dcc1044f0 2023-07-20T02:33:13.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.474+0000 7f8dd1714700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8db8038670 0x7f8db803ab30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8006fd0 tx=0x7f8dc8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:13.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.479+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8dbc018530 con 0x7f8dcc1044f0 2023-07-20T02:33:13.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:10.944+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f8dcc108120 con 0x7f8dcc1044f0 2023-07-20T02:33:13.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.327+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f8dbc02d910 con 0x7f8dcc1044f0 2023-07-20T02:33:13.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.328+0000 7f8dc2ffd700 1 -- 172.21.15.17:0/595143586 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f8dbc04c190 con 0x7f8dcc1044f0 2023-07-20T02:33:13.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8db8038670 msgr2=0x7f8db803ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8db8038670 0x7f8db803ab30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8006fd0 tx=0x7f8dc8006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 msgr2=0x7f8dcc199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc199ef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009fb0 tx=0x7f8dbc011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 shutdown_connections 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7f8db8038670 0x7f8db803ab30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 --2- 172.21.15.17:0/595143586 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8dcc1044f0 0x7f8dcc199ef0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.331+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 >> 172.21.15.17:0/595143586 conn(0x7f8dcc0ffe10 msgr2=0x7f8dcc101500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:13.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.332+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 shutdown_connections 2023-07-20T02:33:13.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:11.332+0000 7f8dd4179700 1 -- 172.21.15.17:0/595143586 wait complete. 2023-07-20T02:33:14.192 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:14.021+0000 7f3385e41200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:33:14.623 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:14.214+0000 7f3385e41200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:33:15.442 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:15.186+0000 7f3385e41200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:33:15.443 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:15.299+0000 7f3385e41200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:33:15.864 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:15.523+0000 7f3385e41200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:33:16.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:15 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1511109486' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T02:33:16.136 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:15.874+0000 7f3385e41200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:33:16.442 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:16.146+0000 7f3385e41200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:33:16.443 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:16.333+0000 7f3385e41200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:33:16.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:33:16.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-20T02:33:16.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T02:33:16.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T02:33:16.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T02:33:16.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:33:16.496 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.421+0000 7fcf788d9700 1 Processor -- start 2023-07-20T02:33:16.496 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.421+0000 7fcf788d9700 1 -- start start 2023-07-20T02:33:16.496 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf788d9700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf741090e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:16.496 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf788d9700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf74073f30 con 0x7fcf74106cf0 2023-07-20T02:33:16.496 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf7259c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf741090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:16.496 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf7259c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf741090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42344/0 (socket says 172.21.15.17:42344) 2023-07-20T02:33:16.496 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf7259c700 1 -- 172.21.15.17:0/3203378166 learned_addr learned my addr 172.21.15.17:0/3203378166 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:16.497 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf7259c700 1 -- 172.21.15.17:0/3203378166 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74109620 con 0x7fcf74106cf0 2023-07-20T02:33:16.497 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf7259c700 1 --2- 172.21.15.17:0/3203378166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf741090e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf640098a0 tx=0x7fcf64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d95ae50a578366a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:16.497 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf71d9b700 1 -- 172.21.15.17:0/3203378166 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf64004160 con 0x7fcf74106cf0 2023-07-20T02:33:16.497 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf71d9b700 1 -- 172.21.15.17:0/3203378166 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf640042c0 con 0x7fcf74106cf0 2023-07-20T02:33:16.497 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.422+0000 7fcf71d9b700 1 -- 172.21.15.17:0/3203378166 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf64011a00 con 0x7fcf74106cf0 2023-07-20T02:33:16.497 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.423+0000 7fcf788d9700 1 -- 172.21.15.17:0/3203378166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 msgr2=0x7fcf741090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:16.497 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.423+0000 7fcf788d9700 1 --2- 172.21.15.17:0/3203378166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf741090e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf640098a0 tx=0x7fcf64009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.423+0000 7fcf788d9700 1 -- 172.21.15.17:0/3203378166 shutdown_connections 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.423+0000 7fcf788d9700 1 --2- 172.21.15.17:0/3203378166 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf741090e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.423+0000 7fcf788d9700 1 -- 172.21.15.17:0/3203378166 >> 172.21.15.17:0/3203378166 conn(0x7fcf74100680 msgr2=0x7fcf74102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.423+0000 7fcf788d9700 1 -- 172.21.15.17:0/3203378166 shutdown_connections 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.423+0000 7fcf788d9700 1 -- 172.21.15.17:0/3203378166 wait complete. 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.424+0000 7fcf788d9700 1 Processor -- start 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.424+0000 7fcf788d9700 1 -- start start 2023-07-20T02:33:16.498 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf788d9700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf74199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:16.499 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf788d9700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7419a440 con 0x7fcf74106cf0 2023-07-20T02:33:16.499 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf7259c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf74199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:16.499 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf7259c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf74199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42346/0 (socket says 172.21.15.17:42346) 2023-07-20T02:33:16.499 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf7259c700 1 -- 172.21.15.17:0/1511109486 learned_addr learned my addr 172.21.15.17:0/1511109486 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:16.499 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf7259c700 1 -- 172.21.15.17:0/1511109486 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf64009550 con 0x7fcf74106cf0 2023-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf7259c700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf74199f00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009fb0 tx=0x7fcf64011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf6bfff700 1 -- 172.21.15.17:0/1511109486 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf64003a40 con 0x7fcf74106cf0 2023-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7419a640 con 0x7fcf74106cf0 2023-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf6bfff700 1 -- 172.21.15.17:0/1511109486 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf64003ba0 con 0x7fcf74106cf0 2023-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.425+0000 7fcf6bfff700 1 -- 172.21.15.17:0/1511109486 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf6401a750 con 0x7fcf74106cf0 2023-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.426+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7419aa60 con 0x7fcf74106cf0 2023-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.427+0000 7fcf6bfff700 1 -- 172.21.15.17:0/1511109486 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fcf6401a9d0 con 0x7fcf74106cf0 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.427+0000 7fcf6bfff700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 0x7fcf6003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.427+0000 7fcf6bfff700 1 -- 172.21.15.17:0/1511109486 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcf6404c3e0 con 0x7fcf74106cf0 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.427+0000 7fcf69d9b700 1 -- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 msgr2=0x7fcf6003ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/1718815307 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.427+0000 7fcf69d9b700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 0x7fcf6003ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.427+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf74061da0 con 0x7fcf74106cf0 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.432+0000 7fcf6bfff700 1 -- 172.21.15.17:0/1511109486 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcf64023390 con 0x7fcf74106cf0 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.628+0000 7fcf69d9b700 1 -- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 msgr2=0x7fcf6003ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/1718815307 2023-07-20T02:33:16.501 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.628+0000 7fcf69d9b700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 0x7fcf6003ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:33:16.502 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.901+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fcf7419d4a0 con 0x7fcf74106cf0 2023-07-20T02:33:16.502 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.901+0000 7fcf6bfff700 1 -- 172.21.15.17:0/1511109486 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fcf64018b40 con 0x7fcf74106cf0 2023-07-20T02:33:16.502 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 msgr2=0x7fcf6003ab80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:33:16.502 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 0x7fcf6003ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:16.502 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 msgr2=0x7fcf74199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:16.502 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf74199f00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009fb0 tx=0x7fcf64011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:16.502 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 shutdown_connections 2023-07-20T02:33:16.503 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcf600386c0 0x7fcf6003ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:16.503 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 --2- 172.21.15.17:0/1511109486 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcf74106cf0 0x7fcf74199f00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:16.503 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.903+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 >> 172.21.15.17:0/1511109486 conn(0x7fcf74100680 msgr2=0x7fcf74101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:16.503 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.904+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 shutdown_connections 2023-07-20T02:33:16.503 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:15.904+0000 7fcf788d9700 1 -- 172.21.15.17:0/1511109486 wait complete. 2023-07-20T02:33:16.503 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for the mgr to restart... 2023-07-20T02:33:16.503 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr epoch 5... 2023-07-20T02:33:16.749 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:16.666+0000 7f3385e41200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:33:17.893 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:17 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:17.501+0000 7f3385e41200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:33:17.894 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:17 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:17.658+0000 7f3385e41200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:33:18.192 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:17 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:17.904+0000 7f3385e41200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:33:18.193 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:18 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:18.041+0000 7f3385e41200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:33:18.692 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:18 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:18.312+0000 7f3385e41200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:33:19.692 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:19 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:19.352+0000 7f3385e41200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:33:20.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: Active manager daemon x restarted 2023-07-20T02:33:20.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: Activating manager daemon x 2023-07-20T02:33:20.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: osdmap e2: 0 total, 0 up, 0 in 2023-07-20T02:33:20.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: mgrmap e6: x(active, starting, since 0.00320189s) 2023-07-20T02:33:20.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:33:20.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T02:33:20.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T02:33:20.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T02:33:20.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T02:33:20.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: Manager daemon x is now available 2023-07-20T02:33:20.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:20.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:20.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:20.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:20.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:20.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T02:33:20.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T02:33:20.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:20.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:20.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:20 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:20.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:33:20.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-20T02:33:20.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T02:33:20.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:33:20.747 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.977+0000 7fcc355e0700 1 Processor -- start 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.977+0000 7fcc355e0700 1 -- start start 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.977+0000 7fcc355e0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30109090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc355e0700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30079e00 con 0x7fcc30106ca0 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30109090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42352/0 (socket says 172.21.15.17:42352) 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2effd700 1 -- 172.21.15.17:0/1269310907 learned_addr learned my addr 172.21.15.17:0/1269310907 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2effd700 1 -- 172.21.15.17:0/1269310907 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc301095d0 con 0x7fcc30106ca0 2023-07-20T02:33:20.748 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2effd700 1 --2- 172.21.15.17:0/1269310907 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30109090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcc180098a0 tx=0x7fcc18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd6a6e63777020c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:20.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2dffb700 1 -- 172.21.15.17:0/1269310907 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc18004160 con 0x7fcc30106ca0 2023-07-20T02:33:20.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2dffb700 1 -- 172.21.15.17:0/1269310907 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc180042c0 con 0x7fcc30106ca0 2023-07-20T02:33:20.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.978+0000 7fcc2dffb700 1 -- 172.21.15.17:0/1269310907 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc18011a00 con 0x7fcc30106ca0 2023-07-20T02:33:20.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.979+0000 7fcc355e0700 1 -- 172.21.15.17:0/1269310907 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 msgr2=0x7fcc30109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:20.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.979+0000 7fcc355e0700 1 --2- 172.21.15.17:0/1269310907 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30109090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcc180098a0 tx=0x7fcc18009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:20.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.979+0000 7fcc355e0700 1 -- 172.21.15.17:0/1269310907 shutdown_connections 2023-07-20T02:33:20.749 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.979+0000 7fcc355e0700 1 --2- 172.21.15.17:0/1269310907 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30109090 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.979+0000 7fcc355e0700 1 -- 172.21.15.17:0/1269310907 >> 172.21.15.17:0/1269310907 conn(0x7fcc301006b0 msgr2=0x7fcc30102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.979+0000 7fcc355e0700 1 -- 172.21.15.17:0/1269310907 shutdown_connections 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.979+0000 7fcc355e0700 1 -- 172.21.15.17:0/1269310907 wait complete. 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.980+0000 7fcc355e0700 1 Processor -- start 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.980+0000 7fcc355e0700 1 -- start start 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.980+0000 7fcc355e0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.980+0000 7fcc355e0700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30106b40 con 0x7fcc30106ca0 2023-07-20T02:33:20.750 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc2effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:20.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc2effd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42354/0 (socket says 172.21.15.17:42354) 2023-07-20T02:33:20.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc2effd700 1 -- 172.21.15.17:0/3254135711 learned_addr learned my addr 172.21.15.17:0/3254135711 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:20.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc2effd700 1 -- 172.21.15.17:0/3254135711 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc18009550 con 0x7fcc30106ca0 2023-07-20T02:33:20.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc2effd700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30106600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009130 tx=0x7fcc18011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:20.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc18003b70 con 0x7fcc30106ca0 2023-07-20T02:33:20.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc18003cd0 con 0x7fcc30106ca0 2023-07-20T02:33:20.751 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc30104c50 con 0x7fcc30106ca0 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc180197d0 con 0x7fcc30106ca0 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.981+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc30105070 con 0x7fcc30106ca0 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.983+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fcc1801abb0 con 0x7fcc30106ca0 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.983+0000 7fcc27fff700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 0x7fcc1c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.983+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 --> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc1c03b2e0 con 0x7fcc1c038710 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.983+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcc1804c910 con 0x7fcc30106ca0 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.983+0000 7fcc2e7fc700 1 -- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 msgr2=0x7fcc1c03abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/1718815307 2023-07-20T02:33:20.752 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:16.983+0000 7fcc2e7fc700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 0x7fcc1c03abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:33:20.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:17.184+0000 7fcc2e7fc700 1 -- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 msgr2=0x7fcc1c03abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/1718815307 2023-07-20T02:33:20.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:17.184+0000 7fcc2e7fc700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 0x7fcc1c03abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:33:20.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:17.585+0000 7fcc2e7fc700 1 -- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 msgr2=0x7fcc1c03abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/1718815307 2023-07-20T02:33:20.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:17.585+0000 7fcc2e7fc700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 0x7fcc1c03abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T02:33:20.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:18.386+0000 7fcc2e7fc700 1 -- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 msgr2=0x7fcc1c03abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/1718815307 2023-07-20T02:33:20.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:18.386+0000 7fcc2e7fc700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 0x7fcc1c03abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T02:33:20.753 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:19.368+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fcc1804b750 con 0x7fcc30106ca0 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:19.368+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 msgr2=0x7fcc1c03abd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:19.368+0000 7fcc27fff700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 0x7fcc1c03abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.369+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fcc18019d90 con 0x7fcc30106ca0 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.369+0000 7fcc27fff700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fcc1c03bff0 0x7fcc1c03e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.370+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc1c03b2e0 con 0x7fcc1c03bff0 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.370+0000 7fcc2e7fc700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fcc1c03bff0 0x7fcc1c03e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.371+0000 7fcc2e7fc700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fcc1c03bff0 0x7fcc1c03e3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcc20003a10 tx=0x7fcc20009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:20.754 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.372+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fcc1c03b2e0 con 0x7fcc1c03bff0 2023-07-20T02:33:20.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.383+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcc301061a0 con 0x7fcc1c03bff0 2023-07-20T02:33:20.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.383+0000 7fcc27fff700 1 -- 172.21.15.17:0/3254135711 <== mgr.14120 v2:172.21.15.17:6800/3625702005 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcc301061a0 con 0x7fcc1c03bff0 2023-07-20T02:33:20.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fcc1c03bff0 msgr2=0x7fcc1c03e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:20.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fcc1c03bff0 0x7fcc1c03e3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcc20003a10 tx=0x7fcc20009290 comp rx=0 tx=0).stop 2023-07-20T02:33:20.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 msgr2=0x7fcc30106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:20.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30106600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009130 tx=0x7fcc18011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:20.755 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 shutdown_connections 2023-07-20T02:33:20.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fcc1c03bff0 0x7fcc1c03e3e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:20.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcc30106ca0 0x7fcc30106600 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:20.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 --2- 172.21.15.17:0/3254135711 >> [v2:172.21.15.17:6800/1718815307,v1:172.21.15.17:6801/1718815307] conn(0x7fcc1c038710 0x7fcc1c03abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:20.756 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 >> 172.21.15.17:0/3254135711 conn(0x7fcc301006b0 msgr2=0x7fcc30101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:20.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 shutdown_connections 2023-07-20T02:33:20.757 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:20.384+0000 7fcc355e0700 1 -- 172.21.15.17:0/3254135711 wait complete. 2023-07-20T02:33:20.757 INFO:teuthology.orchestra.run.smithi017.stdout:mgr epoch 5 is available 2023-07-20T02:33:20.757 INFO:teuthology.orchestra.run.smithi017.stdout:Setting orchestrator backend to cephadm... 2023-07-20T02:33:21.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:21 smithi017 ceph-mon[127345]: Found migration_current of "None". Setting to last migration. 2023-07-20T02:33:21.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:21 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:19] ENGINE Bus STARTING 2023-07-20T02:33:21.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:21 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:20] ENGINE Serving on https://172.21.15.17:7150 2023-07-20T02:33:21.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:21 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:20] ENGINE Serving on http://172.21.15.17:8765 2023-07-20T02:33:21.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:21 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:20] ENGINE Bus STARTED 2023-07-20T02:33:21.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:21 smithi017 ceph-mon[127345]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T02:33:21.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:21 smithi017 ceph-mon[127345]: mgrmap e7: x(active, since 1.00612s) 2023-07-20T02:33:22.096 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.309+0000 7fce2d7f5700 1 Processor -- start 2023-07-20T02:33:22.096 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.309+0000 7fce2d7f5700 1 -- start start 2023-07-20T02:33:22.096 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce2d7f5700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28104920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:22.096 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce2d7f5700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce28104ef0 con 0x7fce28104500 2023-07-20T02:33:22.097 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce26ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28104920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:22.097 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce26ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28104920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42402/0 (socket says 172.21.15.17:42402) 2023-07-20T02:33:22.097 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce26ffd700 1 -- 172.21.15.17:0/2110220207 learned_addr learned my addr 172.21.15.17:0/2110220207 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:22.097 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce26ffd700 1 -- 172.21.15.17:0/2110220207 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28107490 con 0x7fce28104500 2023-07-20T02:33:22.097 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce26ffd700 1 --2- 172.21.15.17:0/2110220207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28104920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce180098a0 tx=0x7fce18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2d56c7ce874198 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:22.097 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce25ffb700 1 -- 172.21.15.17:0/2110220207 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce18004160 con 0x7fce28104500 2023-07-20T02:33:22.098 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce25ffb700 1 -- 172.21.15.17:0/2110220207 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce180042c0 con 0x7fce28104500 2023-07-20T02:33:22.098 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.310+0000 7fce25ffb700 1 -- 172.21.15.17:0/2110220207 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce18011a00 con 0x7fce28104500 2023-07-20T02:33:22.098 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.311+0000 7fce2d7f5700 1 -- 172.21.15.17:0/2110220207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 msgr2=0x7fce28104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:22.098 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.311+0000 7fce2d7f5700 1 --2- 172.21.15.17:0/2110220207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28104920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce180098a0 tx=0x7fce18009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:22.098 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.311+0000 7fce2d7f5700 1 -- 172.21.15.17:0/2110220207 shutdown_connections 2023-07-20T02:33:22.099 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.311+0000 7fce2d7f5700 1 --2- 172.21.15.17:0/2110220207 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28104920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:22.099 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.311+0000 7fce2d7f5700 1 -- 172.21.15.17:0/2110220207 >> 172.21.15.17:0/2110220207 conn(0x7fce280ffe10 msgr2=0x7fce28102270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:22.099 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.311+0000 7fce2d7f5700 1 -- 172.21.15.17:0/2110220207 shutdown_connections 2023-07-20T02:33:22.099 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.311+0000 7fce2d7f5700 1 -- 172.21.15.17:0/2110220207 wait complete. 2023-07-20T02:33:22.100 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce2d7f5700 1 Processor -- start 2023-07-20T02:33:22.100 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce2d7f5700 1 -- start start 2023-07-20T02:33:22.100 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce2d7f5700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:22.100 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce2d7f5700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2819a490 con 0x7fce28104500 2023-07-20T02:33:22.101 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce26ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:22.101 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce26ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42404/0 (socket says 172.21.15.17:42404) 2023-07-20T02:33:22.101 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce26ffd700 1 -- 172.21.15.17:0/3987128726 learned_addr learned my addr 172.21.15.17:0/3987128726 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:22.101 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.312+0000 7fce26ffd700 1 -- 172.21.15.17:0/3987128726 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18009550 con 0x7fce28104500 2023-07-20T02:33:22.102 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.313+0000 7fce26ffd700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28199f50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce18009fb0 tx=0x7fce18011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:22.102 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.313+0000 7fce1ffff700 1 -- 172.21.15.17:0/3987128726 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce18003a40 con 0x7fce28104500 2023-07-20T02:33:22.102 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.313+0000 7fce1ffff700 1 -- 172.21.15.17:0/3987128726 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce18003ba0 con 0x7fce28104500 2023-07-20T02:33:22.102 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.313+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2819a690 con 0x7fce28104500 2023-07-20T02:33:22.103 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.313+0000 7fce1ffff700 1 -- 172.21.15.17:0/3987128726 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce1801a750 con 0x7fce28104500 2023-07-20T02:33:22.103 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.313+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2819aab0 con 0x7fce28104500 2023-07-20T02:33:22.103 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.314+0000 7fce1ffff700 1 -- 172.21.15.17:0/3987128726 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fce1801a9d0 con 0x7fce28104500 2023-07-20T02:33:22.103 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.314+0000 7fce1ffff700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fce140385a0 0x7fce1403aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:22.103 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.314+0000 7fce1ffff700 1 -- 172.21.15.17:0/3987128726 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fce1804c570 con 0x7fce28104500 2023-07-20T02:33:22.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.314+0000 7fce267fc700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fce140385a0 0x7fce1403aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:22.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.314+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce28061da0 con 0x7fce28104500 2023-07-20T02:33:22.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.315+0000 7fce267fc700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fce140385a0 0x7fce1403aa60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:22.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.319+0000 7fce1ffff700 1 -- 172.21.15.17:0/3987128726 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce18023390 con 0x7fce28104500 2023-07-20T02:33:22.105 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.702+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fce28102170 con 0x7fce140385a0 2023-07-20T02:33:22.105 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.707+0000 7fce1ffff700 1 -- 172.21.15.17:0/3987128726 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fce28102170 con 0x7fce140385a0 2023-07-20T02:33:22.105 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fce140385a0 msgr2=0x7fce1403aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:22.105 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fce140385a0 0x7fce1403aa60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:22.106 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 msgr2=0x7fce28199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:22.106 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28199f50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce18009fb0 tx=0x7fce18011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:22.106 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 shutdown_connections 2023-07-20T02:33:22.106 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fce140385a0 0x7fce1403aa60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:22.107 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 --2- 172.21.15.17:0/3987128726 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fce28104500 0x7fce28199f50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:22.107 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 >> 172.21.15.17:0/3987128726 conn(0x7fce280ffe10 msgr2=0x7fce28101550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:22.107 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 shutdown_connections 2023-07-20T02:33:22.107 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:21.710+0000 7fce2d7f5700 1 -- 172.21.15.17:0/3987128726 wait complete. 2023-07-20T02:33:22.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:22 smithi017 ceph-mon[127345]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T02:33:22.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:22 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:22.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:22 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:23.242 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-20T02:33:23.242 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.563+0000 7fb46fed2700 1 Processor -- start 2023-07-20T02:33:23.242 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46fed2700 1 -- start start 2023-07-20T02:33:23.243 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46fed2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468105520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:23.243 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46fed2700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb468105af0 con 0x7fb468105100 2023-07-20T02:33:23.243 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46dc6e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468105520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:23.243 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46dc6e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468105520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42408/0 (socket says 172.21.15.17:42408) 2023-07-20T02:33:23.243 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46dc6e700 1 -- 172.21.15.17:0/2391452304 learned_addr learned my addr 172.21.15.17:0/2391452304 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:23.243 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46dc6e700 1 -- 172.21.15.17:0/2391452304 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb468106300 con 0x7fb468105100 2023-07-20T02:33:23.243 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.564+0000 7fb46dc6e700 1 --2- 172.21.15.17:0/2391452304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468105520 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0098a0 tx=0x7fb45c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8d01430669b318 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:23.244 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46cc6c700 1 -- 172.21.15.17:0/2391452304 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb45c004160 con 0x7fb468105100 2023-07-20T02:33:23.244 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46cc6c700 1 -- 172.21.15.17:0/2391452304 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb45c0042c0 con 0x7fb468105100 2023-07-20T02:33:23.244 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46cc6c700 1 -- 172.21.15.17:0/2391452304 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb45c011a00 con 0x7fb468105100 2023-07-20T02:33:23.244 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46fed2700 1 -- 172.21.15.17:0/2391452304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 msgr2=0x7fb468105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:23.244 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46fed2700 1 --2- 172.21.15.17:0/2391452304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468105520 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0098a0 tx=0x7fb45c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:23.245 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46fed2700 1 -- 172.21.15.17:0/2391452304 shutdown_connections 2023-07-20T02:33:23.245 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46fed2700 1 --2- 172.21.15.17:0/2391452304 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468105520 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:23.245 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46fed2700 1 -- 172.21.15.17:0/2391452304 >> 172.21.15.17:0/2391452304 conn(0x7fb468100680 msgr2=0x7fb468102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:23.245 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46fed2700 1 -- 172.21.15.17:0/2391452304 shutdown_connections 2023-07-20T02:33:23.245 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.565+0000 7fb46fed2700 1 -- 172.21.15.17:0/2391452304 wait complete. 2023-07-20T02:33:23.246 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.566+0000 7fb46fed2700 1 Processor -- start 2023-07-20T02:33:23.246 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.566+0000 7fb46fed2700 1 -- start start 2023-07-20T02:33:23.246 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.566+0000 7fb46fed2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468190290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:23.246 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.566+0000 7fb46fed2700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4681907d0 con 0x7fb468105100 2023-07-20T02:33:23.246 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.566+0000 7fb46dc6e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468190290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.566+0000 7fb46dc6e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468190290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42410/0 (socket says 172.21.15.17:42410) 2023-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.566+0000 7fb46dc6e700 1 -- 172.21.15.17:0/2505606419 learned_addr learned my addr 172.21.15.17:0/2505606419 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.567+0000 7fb46dc6e700 1 -- 172.21.15.17:0/2505606419 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c009550 con 0x7fb468105100 2023-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.567+0000 7fb46dc6e700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468190290 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009130 tx=0x7fb45c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.567+0000 7fb45affd700 1 -- 172.21.15.17:0/2505606419 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb45c003ab0 con 0x7fb468105100 2023-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.567+0000 7fb45affd700 1 -- 172.21.15.17:0/2505606419 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb45c003c10 con 0x7fb468105100 2023-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.567+0000 7fb45affd700 1 -- 172.21.15.17:0/2505606419 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb45c0247d0 con 0x7fb468105100 2023-07-20T02:33:23.248 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.567+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46818e8e0 con 0x7fb468105100 2023-07-20T02:33:23.248 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.567+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46818ed00 con 0x7fb468105100 2023-07-20T02:33:23.248 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.568+0000 7fb45affd700 1 -- 172.21.15.17:0/2505606419 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb45c02d3f0 con 0x7fb468105100 2023-07-20T02:33:23.248 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.568+0000 7fb45affd700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb454038760 0x7fb45403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:23.248 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.568+0000 7fb45affd700 1 -- 172.21.15.17:0/2505606419 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb45c04c830 con 0x7fb468105100 2023-07-20T02:33:23.255 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.568+0000 7fb46d46d700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb454038760 0x7fb45403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:23.256 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.568+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb468061da0 con 0x7fb468105100 2023-07-20T02:33:23.256 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.569+0000 7fb46d46d700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb454038760 0x7fb45403ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb464006fd0 tx=0x7fb464006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:23.256 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.573+0000 7fb45affd700 1 -- 172.21.15.17:0/2505606419 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb45c010970 con 0x7fb468105100 2023-07-20T02:33:23.256 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.930+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb46818f5c0 con 0x7fb454038760 2023-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.931+0000 7fb45affd700 1 -- 172.21.15.17:0/2505606419 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb46818f5c0 con 0x7fb454038760 2023-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb454038760 msgr2=0x7fb45403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb454038760 0x7fb45403ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb464006fd0 tx=0x7fb464006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 msgr2=0x7fb468190290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468190290 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009130 tx=0x7fb45c011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 shutdown_connections 2023-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb454038760 0x7fb45403ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:23.258 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 --2- 172.21.15.17:0/2505606419 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb468105100 0x7fb468190290 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:23.258 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 >> 172.21.15.17:0/2505606419 conn(0x7fb468100680 msgr2=0x7fb468073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:23.258 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 shutdown_connections 2023-07-20T02:33:23.258 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:22.933+0000 7fb46fed2700 1 -- 172.21.15.17:0/2505606419 wait complete. 2023-07-20T02:33:23.258 INFO:teuthology.orchestra.run.smithi017.stdout:Generating ssh key... 2023-07-20T02:33:23.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:23 smithi017 ceph-mon[127345]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:23.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:23 smithi017 ceph-mon[127345]: mgrmap e8: x(active, since 3s) 2023-07-20T02:33:24.630 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-mon[127345]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:24.630 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:24.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:24.631 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: Generating public/private rsa key pair. 2023-07-20T02:33:24.631 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: Your identification has been saved in /tmp/tmpvcn4cm8j/key. 2023-07-20T02:33:24.632 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: Your public key has been saved in /tmp/tmpvcn4cm8j/key.pub. 2023-07-20T02:33:24.632 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: The key fingerprint is: 2023-07-20T02:33:24.632 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: SHA256:FIByJ9+wPGik0atJtdMGRDtTQuVm+8KZxH7zVBAOZTw ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:33:24.633 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: The key's randomart image is: 2023-07-20T02:33:24.633 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: +---[RSA 3072]----+ 2023-07-20T02:33:24.633 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | ==o+.oo+ | 2023-07-20T02:33:24.634 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | o O=o =E. | 2023-07-20T02:33:24.634 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | B+@++. o. | 2023-07-20T02:33:24.634 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | o **Bo. . | 2023-07-20T02:33:24.635 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | . + o+.S . | 2023-07-20T02:33:24.635 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | o + + . | 2023-07-20T02:33:24.635 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | * + . | 2023-07-20T02:33:24.636 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | o + | 2023-07-20T02:33:24.636 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: | . | 2023-07-20T02:33:24.636 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: +----[SHA256]-----+ 2023-07-20T02:33:24.688 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb483039700 1 Processor -- start 2023-07-20T02:33:24.688 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb483039700 1 -- start start 2023-07-20T02:33:24.688 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb483039700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c109060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:24.689 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb483039700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c073fb0 con 0x7fb47c106c70 2023-07-20T02:33:24.689 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb480dd5700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c109060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:24.689 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb480dd5700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c109060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42414/0 (socket says 172.21.15.17:42414) 2023-07-20T02:33:24.689 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb480dd5700 1 -- 172.21.15.17:0/543466377 learned_addr learned my addr 172.21.15.17:0/543466377 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:24.689 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.708+0000 7fb480dd5700 1 -- 172.21.15.17:0/543466377 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c1095a0 con 0x7fb47c106c70 2023-07-20T02:33:24.689 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb480dd5700 1 --2- 172.21.15.17:0/543466377 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c109060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb4700098a0 tx=0x7fb470009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef060e673135e1c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:24.689 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb47b7fe700 1 -- 172.21.15.17:0/543466377 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb470004160 con 0x7fb47c106c70 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb47b7fe700 1 -- 172.21.15.17:0/543466377 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4700042c0 con 0x7fb47c106c70 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb47b7fe700 1 -- 172.21.15.17:0/543466377 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb470011a00 con 0x7fb47c106c70 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb483039700 1 -- 172.21.15.17:0/543466377 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 msgr2=0x7fb47c109060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb483039700 1 --2- 172.21.15.17:0/543466377 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c109060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb4700098a0 tx=0x7fb470009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb483039700 1 -- 172.21.15.17:0/543466377 shutdown_connections 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb483039700 1 --2- 172.21.15.17:0/543466377 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c109060 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb483039700 1 -- 172.21.15.17:0/543466377 >> 172.21.15.17:0/543466377 conn(0x7fb47c100680 msgr2=0x7fb47c102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:24.690 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.709+0000 7fb483039700 1 -- 172.21.15.17:0/543466377 shutdown_connections 2023-07-20T02:33:24.691 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb483039700 1 -- 172.21.15.17:0/543466377 wait complete. 2023-07-20T02:33:24.691 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb483039700 1 Processor -- start 2023-07-20T02:33:24.691 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb483039700 1 -- start start 2023-07-20T02:33:24.691 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb483039700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:24.691 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb483039700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19a470 con 0x7fb47c106c70 2023-07-20T02:33:24.691 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb480dd5700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:24.691 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb480dd5700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42416/0 (socket says 172.21.15.17:42416) 2023-07-20T02:33:24.692 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.710+0000 7fb480dd5700 1 -- 172.21.15.17:0/3233649378 learned_addr learned my addr 172.21.15.17:0/3233649378 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:24.692 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.711+0000 7fb480dd5700 1 -- 172.21.15.17:0/3233649378 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb470009550 con 0x7fb47c106c70 2023-07-20T02:33:24.692 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.711+0000 7fb480dd5700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c199f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb470009130 tx=0x7fb470011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:24.692 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.711+0000 7fb479ffb700 1 -- 172.21.15.17:0/3233649378 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb470003b70 con 0x7fb47c106c70 2023-07-20T02:33:24.692 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.711+0000 7fb479ffb700 1 -- 172.21.15.17:0/3233649378 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb470003cd0 con 0x7fb47c106c70 2023-07-20T02:33:24.692 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.711+0000 7fb479ffb700 1 -- 172.21.15.17:0/3233649378 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4700197d0 con 0x7fb47c106c70 2023-07-20T02:33:24.692 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.711+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c19a670 con 0x7fb47c106c70 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.711+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c19aa90 con 0x7fb47c106c70 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.712+0000 7fb479ffb700 1 -- 172.21.15.17:0/3233649378 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb47001abb0 con 0x7fb47c106c70 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.712+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c10a3a0 con 0x7fb47c106c70 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.713+0000 7fb479ffb700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.713+0000 7fb479ffb700 1 -- 172.21.15.17:0/3233649378 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb47004c8d0 con 0x7fb47c106c70 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.713+0000 7fb47bfff700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.713+0000 7fb47bfff700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb468006fd0 tx=0x7fb468006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:23.717+0000 7fb479ffb700 1 -- 172.21.15.17:0/3233649378 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb47002db40 con 0x7fb47c106c70 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.074+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb47c1029e0 con 0x7fb46c038760 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.309+0000 7fb479ffb700 1 -- 172.21.15.17:0/3233649378 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb47c1029e0 con 0x7fb46c038760 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 msgr2=0x7fb46c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb468006fd0 tx=0x7fb468006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 msgr2=0x7fb47c199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c199f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb470009130 tx=0x7fb470011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 shutdown_connections 2023-07-20T02:33:24.694 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:24.695 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 --2- 172.21.15.17:0/3233649378 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106c70 0x7fb47c199f30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:24.695 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 >> 172.21.15.17:0/3233649378 conn(0x7fb47c100680 msgr2=0x7fb47c101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:24.695 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 shutdown_connections 2023-07-20T02:33:24.695 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:24.311+0000 7fb483039700 1 -- 172.21.15.17:0/3233649378 wait complete. 2023-07-20T02:33:25.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:25 smithi017 ceph-mon[127345]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:25.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:25 smithi017 ceph-mon[127345]: Generating ssh key... 2023-07-20T02:33:25.830 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4PLr4CHxSrpIfvLEMjP/sKT1OFec4vv7Rq5QmHtikMPj5X3pVErPLv5lb5JiUp/EkhA1a1odQXwn32rcwlyidRVBTa3idi2mBejjSkPHZiAamSUFYyoCU6pwgks2Vlwr7Hifeejl9p4tol7PxnKVwa3r9muyNhXdRmjWQx5VMUCQlDG3OWsXjI17iC0qRD02WEzJso9YqUxu8i2xtjQQ51AptEKmx8pVM8a954VI70cwPlM1LVxx+yeD5OuNjLDB62xFxvku8ymiWQL5aY0OdbvzK2/OZenyqJCe4kLzmL6syj2ursYlzz1x3fPtnnykWXp42Y7JVCL6uU9gbICH6Uh1NedX2HUnbe72jW4+hNsDSt0NNmVPiYEtdc3DFQPXqXev3p+De1DYrZs5Nhzci1h1BSDvJD8QgQpb2Y6gAzKNlBc0KhkO4T3uHLXnfYsKCRrtnErsE7YDRvqZbP3cI8oDTNFWZWn7e4+eS59IDRr+OEB6tdHEfc+hNQYAV2oc= ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:33:25.830 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.141+0000 7fb483182700 1 Processor -- start 2023-07-20T02:33:25.832 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.141+0000 7fb483182700 1 -- start start 2023-07-20T02:33:25.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.141+0000 7fb483182700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c1090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:25.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.141+0000 7fb483182700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c073f70 con 0x7fb47c106cb0 2023-07-20T02:33:25.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.141+0000 7fb480f1e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c1090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:25.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.141+0000 7fb480f1e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c1090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42420/0 (socket says 172.21.15.17:42420) 2023-07-20T02:33:25.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.141+0000 7fb480f1e700 1 -- 172.21.15.17:0/1440892064 learned_addr learned my addr 172.21.15.17:0/1440892064 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:25.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.142+0000 7fb480f1e700 1 -- 172.21.15.17:0/1440892064 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c1095e0 con 0x7fb47c106cb0 2023-07-20T02:33:25.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.142+0000 7fb480f1e700 1 --2- 172.21.15.17:0/1440892064 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c1090a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb4700098a0 tx=0x7fb470009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527246484f1c37 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:25.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.142+0000 7fb47b7fe700 1 -- 172.21.15.17:0/1440892064 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb470004160 con 0x7fb47c106cb0 2023-07-20T02:33:25.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.142+0000 7fb47b7fe700 1 -- 172.21.15.17:0/1440892064 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4700042c0 con 0x7fb47c106cb0 2023-07-20T02:33:25.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.142+0000 7fb47b7fe700 1 -- 172.21.15.17:0/1440892064 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb470011a00 con 0x7fb47c106cb0 2023-07-20T02:33:25.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 -- 172.21.15.17:0/1440892064 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 msgr2=0x7fb47c1090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:25.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 --2- 172.21.15.17:0/1440892064 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c1090a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb4700098a0 tx=0x7fb470009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:25.837 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 -- 172.21.15.17:0/1440892064 shutdown_connections 2023-07-20T02:33:25.837 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 --2- 172.21.15.17:0/1440892064 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c1090a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:25.837 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 -- 172.21.15.17:0/1440892064 >> 172.21.15.17:0/1440892064 conn(0x7fb47c100680 msgr2=0x7fb47c102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:25.837 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 -- 172.21.15.17:0/1440892064 shutdown_connections 2023-07-20T02:33:25.838 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 -- 172.21.15.17:0/1440892064 wait complete. 2023-07-20T02:33:25.838 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.143+0000 7fb483182700 1 Processor -- start 2023-07-20T02:33:25.838 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb483182700 1 -- start start 2023-07-20T02:33:25.839 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb483182700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:25.839 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb483182700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19a480 con 0x7fb47c106cb0 2023-07-20T02:33:25.839 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb480f1e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:25.840 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb480f1e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42422/0 (socket says 172.21.15.17:42422) 2023-07-20T02:33:25.840 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb480f1e700 1 -- 172.21.15.17:0/3149367538 learned_addr learned my addr 172.21.15.17:0/3149367538 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:25.840 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb480f1e700 1 -- 172.21.15.17:0/3149367538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb470009550 con 0x7fb47c106cb0 2023-07-20T02:33:25.840 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb480f1e700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c199f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb470009130 tx=0x7fb470011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:25.841 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.144+0000 7fb479ffb700 1 -- 172.21.15.17:0/3149367538 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb470003b70 con 0x7fb47c106cb0 2023-07-20T02:33:25.841 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.145+0000 7fb479ffb700 1 -- 172.21.15.17:0/3149367538 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb470003cd0 con 0x7fb47c106cb0 2023-07-20T02:33:25.841 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.145+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c19a680 con 0x7fb47c106cb0 2023-07-20T02:33:25.842 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.145+0000 7fb479ffb700 1 -- 172.21.15.17:0/3149367538 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4700197d0 con 0x7fb47c106cb0 2023-07-20T02:33:25.842 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.145+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c19aaa0 con 0x7fb47c106cb0 2023-07-20T02:33:25.842 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.146+0000 7fb479ffb700 1 -- 172.21.15.17:0/3149367538 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb47001abb0 con 0x7fb47c106cb0 2023-07-20T02:33:25.842 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.146+0000 7fb479ffb700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:25.843 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.146+0000 7fb479ffb700 1 -- 172.21.15.17:0/3149367538 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb47004c8e0 con 0x7fb47c106cb0 2023-07-20T02:33:25.843 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.146+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c061da0 con 0x7fb47c106cb0 2023-07-20T02:33:25.843 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.146+0000 7fb47bfff700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:25.844 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.146+0000 7fb47bfff700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb468006fd0 tx=0x7fb468006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:25.844 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.151+0000 7fb479ffb700 1 -- 172.21.15.17:0/3149367538 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb47001a430 con 0x7fb47c106cb0 2023-07-20T02:33:25.844 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.506+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb47c1029f0 con 0x7fb46c038760 2023-07-20T02:33:25.845 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.507+0000 7fb479ffb700 1 -- 172.21.15.17:0/3149367538 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb47c1029f0 con 0x7fb46c038760 2023-07-20T02:33:25.845 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 msgr2=0x7fb46c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:25.845 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb468006fd0 tx=0x7fb468006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:25.845 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 msgr2=0x7fb47c199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:25.846 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c199f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb470009130 tx=0x7fb470011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:25.846 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 shutdown_connections 2023-07-20T02:33:25.846 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb46c038760 0x7fb46c03ac20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:25.847 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 --2- 172.21.15.17:0/3149367538 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb47c106cb0 0x7fb47c199f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:25.847 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 >> 172.21.15.17:0/3149367538 conn(0x7fb47c100680 msgr2=0x7fb47c101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:25.847 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 shutdown_connections 2023-07-20T02:33:25.847 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:25.509+0000 7fb483182700 1 -- 172.21.15.17:0/3149367538 wait complete. 2023-07-20T02:33:25.848 INFO:teuthology.orchestra.run.smithi017.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:33:25.848 INFO:teuthology.orchestra.run.smithi017.stdout:Adding key to root@localhost authorized_keys... 2023-07-20T02:33:25.849 INFO:teuthology.orchestra.run.smithi017.stdout:Adding host smithi017... 2023-07-20T02:33:26.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:26 smithi017 ceph-mon[127345]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:27.505 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:27 smithi017 ceph-mon[127345]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi017", "addr": "172.21.15.17", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:27.506 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:27 smithi017 ceph-mon[127345]: Deploying cephadm binary to smithi017 2023-07-20T02:33:29.784 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout Added host 'smithi017' with addr '172.21.15.17' 2023-07-20T02:33:29.785 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab3a7b700 1 Processor -- start 2023-07-20T02:33:29.785 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab3a7b700 1 -- start start 2023-07-20T02:33:29.785 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab3a7b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac1090e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:29.785 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab3a7b700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac073f30 con 0x7f8aac106cf0 2023-07-20T02:33:29.785 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab1817700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac1090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:29.785 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab1817700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac1090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42426/0 (socket says 172.21.15.17:42426) 2023-07-20T02:33:29.786 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab1817700 1 -- 172.21.15.17:0/3403194672 learned_addr learned my addr 172.21.15.17:0/3403194672 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:29.786 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab1817700 1 -- 172.21.15.17:0/3403194672 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac109620 con 0x7f8aac106cf0 2023-07-20T02:33:29.786 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.337+0000 7f8ab1817700 1 --2- 172.21.15.17:0/3403194672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac1090e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80098a0 tx=0x7f8aa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc67c0e52db6535a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:29.786 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab0815700 1 -- 172.21.15.17:0/3403194672 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa8004160 con 0x7f8aac106cf0 2023-07-20T02:33:29.786 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab0815700 1 -- 172.21.15.17:0/3403194672 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa80042c0 con 0x7f8aac106cf0 2023-07-20T02:33:29.786 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab0815700 1 -- 172.21.15.17:0/3403194672 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa8011a00 con 0x7f8aac106cf0 2023-07-20T02:33:29.786 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3403194672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 msgr2=0x7f8aac1090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab3a7b700 1 --2- 172.21.15.17:0/3403194672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac1090e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80098a0 tx=0x7f8aa8009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3403194672 shutdown_connections 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab3a7b700 1 --2- 172.21.15.17:0/3403194672 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac1090e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.338+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3403194672 >> 172.21.15.17:0/3403194672 conn(0x7f8aac100680 msgr2=0x7f8aac102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3403194672 shutdown_connections 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3403194672 wait complete. 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab3a7b700 1 Processor -- start 2023-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab3a7b700 1 -- start start 2023-07-20T02:33:29.788 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab3a7b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:29.788 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab3a7b700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac19a440 con 0x7f8aac106cf0 2023-07-20T02:33:29.788 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab1817700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:29.788 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab1817700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42428/0 (socket says 172.21.15.17:42428) 2023-07-20T02:33:29.788 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.339+0000 7f8ab1817700 1 -- 172.21.15.17:0/3834857763 learned_addr learned my addr 172.21.15.17:0/3834857763 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:29.788 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.340+0000 7f8ab1817700 1 -- 172.21.15.17:0/3834857763 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8009550 con 0x7f8aac106cf0 2023-07-20T02:33:29.788 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.340+0000 7f8ab1817700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac199f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8009fb0 tx=0x7f8aa8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.340+0000 7f8a9effd700 1 -- 172.21.15.17:0/3834857763 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa8003a40 con 0x7f8aac106cf0 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.340+0000 7f8a9effd700 1 -- 172.21.15.17:0/3834857763 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa8003ba0 con 0x7f8aac106cf0 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.340+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac19a640 con 0x7f8aac106cf0 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.340+0000 7f8a9effd700 1 -- 172.21.15.17:0/3834857763 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa801a750 con 0x7f8aac106cf0 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.340+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac19aa60 con 0x7f8aac106cf0 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.341+0000 7f8a9effd700 1 -- 172.21.15.17:0/3834857763 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f8aa801a9d0 con 0x7f8aac106cf0 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.341+0000 7f8a9effd700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f8a980386c0 0x7f8a9803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:29.789 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.341+0000 7f8a9effd700 1 -- 172.21.15.17:0/3834857763 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8aa804c930 con 0x7f8aac106cf0 2023-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.341+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac061da0 con 0x7f8aac106cf0 2023-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.341+0000 7f8ab1016700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f8a980386c0 0x7f8a9803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.342+0000 7f8ab1016700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f8a980386c0 0x7f8a9803ab80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0006fd0 tx=0x7f8aa0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.346+0000 7f8a9effd700 1 -- 172.21.15.17:0/3834857763 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8aa8023390 con 0x7f8aac106cf0 2023-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:26.711+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi017", "addr": "172.21.15.17", "target": ["mon-mgr", ""]}) v1 -- 0x7f8aac1029c0 con 0x7f8a980386c0 2023-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.450+0000 7f8a9effd700 1 -- 172.21.15.17:0/3834857763 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f8aac1029c0 con 0x7f8a980386c0 2023-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f8a980386c0 msgr2=0x7f8a9803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f8a980386c0 0x7f8a9803ab80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0006fd0 tx=0x7f8aa0006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 msgr2=0x7f8aac199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac199f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8009fb0 tx=0x7f8aa8011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 shutdown_connections 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f8a980386c0 0x7f8a9803ab80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 --2- 172.21.15.17:0/3834857763 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8aac106cf0 0x7f8aac199f00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 >> 172.21.15.17:0/3834857763 conn(0x7f8aac100680 msgr2=0x7f8aac101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.453+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 shutdown_connections 2023-07-20T02:33:29.792 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:29.454+0000 7f8ab3a7b700 1 -- 172.21.15.17:0/3834857763 wait complete. 2023-07-20T02:33:29.792 INFO:teuthology.orchestra.run.smithi017.stdout:Deploying unmanaged mon service... 2023-07-20T02:33:30.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:30 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:30.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:30 smithi017 ceph-mon[127345]: Added host smithi017 2023-07-20T02:33:30.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:30 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:31.127 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-20T02:33:31.131 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.255+0000 7f4fa92e0700 1 Processor -- start 2023-07-20T02:33:31.131 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.255+0000 7f4fa92e0700 1 -- start start 2023-07-20T02:33:31.132 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.255+0000 7f4fa92e0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa4106610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:31.132 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.255+0000 7f4fa92e0700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4106be0 con 0x7f4fa41061f0 2023-07-20T02:33:31.132 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.256+0000 7f4fa2ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa4106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:31.132 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.256+0000 7f4fa2ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa4106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42462/0 (socket says 172.21.15.17:42462) 2023-07-20T02:33:31.132 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.256+0000 7f4fa2ffd700 1 -- 172.21.15.17:0/3898341754 learned_addr learned my addr 172.21.15.17:0/3898341754 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:31.132 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.256+0000 7f4fa2ffd700 1 -- 172.21.15.17:0/3898341754 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa41073f0 con 0x7f4fa41061f0 2023-07-20T02:33:31.132 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.256+0000 7f4fa2ffd700 1 --2- 172.21.15.17:0/3898341754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa4106610 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4f980098a0 tx=0x7f4f98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5ed91748833b00 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:31.133 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa1ffb700 1 -- 172.21.15.17:0/3898341754 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f98004160 con 0x7f4fa41061f0 2023-07-20T02:33:31.133 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa1ffb700 1 -- 172.21.15.17:0/3898341754 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f980042c0 con 0x7f4fa41061f0 2023-07-20T02:33:31.133 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa1ffb700 1 -- 172.21.15.17:0/3898341754 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f98011a00 con 0x7f4fa41061f0 2023-07-20T02:33:31.133 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa92e0700 1 -- 172.21.15.17:0/3898341754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 msgr2=0x7f4fa4106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:31.133 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa92e0700 1 --2- 172.21.15.17:0/3898341754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa4106610 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4f980098a0 tx=0x7f4f98009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:31.133 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa92e0700 1 -- 172.21.15.17:0/3898341754 shutdown_connections 2023-07-20T02:33:31.133 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa92e0700 1 --2- 172.21.15.17:0/3898341754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa4106610 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa92e0700 1 -- 172.21.15.17:0/3898341754 >> 172.21.15.17:0/3898341754 conn(0x7f4fa4101790 msgr2=0x7f4fa4103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa92e0700 1 -- 172.21.15.17:0/3898341754 shutdown_connections 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.257+0000 7f4fa92e0700 1 -- 172.21.15.17:0/3898341754 wait complete. 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.258+0000 7f4fa92e0700 1 Processor -- start 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.258+0000 7f4fa92e0700 1 -- start start 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.258+0000 7f4fa92e0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa407bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.258+0000 7f4fa92e0700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa407c120 con 0x7f4fa41061f0 2023-07-20T02:33:31.134 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.258+0000 7f4fa2ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa407bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.258+0000 7f4fa2ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa407bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42464/0 (socket says 172.21.15.17:42464) 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.258+0000 7f4fa2ffd700 1 -- 172.21.15.17:0/75375986 learned_addr learned my addr 172.21.15.17:0/75375986 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.259+0000 7f4fa2ffd700 1 -- 172.21.15.17:0/75375986 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f98009550 con 0x7f4fa41061f0 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.259+0000 7f4fa2ffd700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa407bbe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009130 tx=0x7f4f98011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.259+0000 7f4f8bfff700 1 -- 172.21.15.17:0/75375986 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f98003ab0 con 0x7f4fa41061f0 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.259+0000 7f4f8bfff700 1 -- 172.21.15.17:0/75375986 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f98003c10 con 0x7f4fa41061f0 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.259+0000 7f4f8bfff700 1 -- 172.21.15.17:0/75375986 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f980247d0 con 0x7f4fa41061f0 2023-07-20T02:33:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.259+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa407a230 con 0x7f4fa41061f0 2023-07-20T02:33:31.136 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.259+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa407a650 con 0x7f4fa41061f0 2023-07-20T02:33:31.136 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.260+0000 7f4f8bfff700 1 -- 172.21.15.17:0/75375986 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f4f9801a430 con 0x7f4fa41061f0 2023-07-20T02:33:31.136 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.260+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa4061da0 con 0x7f4fa41061f0 2023-07-20T02:33:31.136 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.260+0000 7f4f8bfff700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f4f8c0387b0 0x7f4f8c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:31.136 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.260+0000 7f4f8bfff700 1 -- 172.21.15.17:0/75375986 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4f9804c8b0 con 0x7f4fa41061f0 2023-07-20T02:33:31.136 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.260+0000 7f4fa27fc700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f4f8c0387b0 0x7f4f8c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:31.136 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.261+0000 7f4fa27fc700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f4f8c0387b0 0x7f4f8c03ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f94006fd0 tx=0x7f4f94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.265+0000 7f4f8bfff700 1 -- 172.21.15.17:0/75375986 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4f98018b40 con 0x7f4fa41061f0 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.649+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4fa410a5f0 con 0x7f4f8c0387b0 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.654+0000 7f4f8bfff700 1 -- 172.21.15.17:0/75375986 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4fa410a5f0 con 0x7f4f8c0387b0 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.656+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f4f8c0387b0 msgr2=0x7f4f8c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.656+0000 7f4fa92e0700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f4f8c0387b0 0x7f4f8c03ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f94006fd0 tx=0x7f4f94006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.656+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 msgr2=0x7f4fa407bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.656+0000 7f4fa92e0700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa407bbe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009130 tx=0x7f4f98011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:31.137 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.656+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 shutdown_connections 2023-07-20T02:33:31.138 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.656+0000 7f4fa92e0700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f4f8c0387b0 0x7f4f8c03ac70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:31.138 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.656+0000 7f4fa92e0700 1 --2- 172.21.15.17:0/75375986 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4fa41061f0 0x7f4fa407bbe0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:31.138 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.657+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 >> 172.21.15.17:0/75375986 conn(0x7f4fa4101790 msgr2=0x7f4fa4102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:31.138 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.657+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 shutdown_connections 2023-07-20T02:33:31.138 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:30.657+0000 7f4fa92e0700 1 -- 172.21.15.17:0/75375986 wait complete. 2023-07-20T02:33:31.138 INFO:teuthology.orchestra.run.smithi017.stdout:Deploying unmanaged mgr service... 2023-07-20T02:33:31.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:31 smithi017 ceph-mon[127345]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:31.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:31 smithi017 ceph-mon[127345]: Saving service mon spec with placement count:5 2023-07-20T02:33:31.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:31 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:32.482 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-20T02:33:32.482 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.700+0000 7fb58b5d0700 1 Processor -- start 2023-07-20T02:33:32.483 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58b5d0700 1 -- start start 2023-07-20T02:33:32.483 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58b5d0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584105520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:32.483 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58b5d0700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb584105af0 con 0x7fb584105100 2023-07-20T02:33:32.483 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58936c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584105520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:32.483 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58936c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584105520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42468/0 (socket says 172.21.15.17:42468) 2023-07-20T02:33:32.483 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58936c700 1 -- 172.21.15.17:0/3444418329 learned_addr learned my addr 172.21.15.17:0/3444418329 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:32.483 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58936c700 1 -- 172.21.15.17:0/3444418329 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb584106300 con 0x7fb584105100 2023-07-20T02:33:32.484 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.701+0000 7fb58936c700 1 --2- 172.21.15.17:0/3444418329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584105520 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb5800098a0 tx=0x7fb580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4f3cd0db09455a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:32.484 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.702+0000 7fb57bfff700 1 -- 172.21.15.17:0/3444418329 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb580004160 con 0x7fb584105100 2023-07-20T02:33:32.484 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.702+0000 7fb57bfff700 1 -- 172.21.15.17:0/3444418329 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5800042c0 con 0x7fb584105100 2023-07-20T02:33:32.484 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.702+0000 7fb57bfff700 1 -- 172.21.15.17:0/3444418329 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb580011a00 con 0x7fb584105100 2023-07-20T02:33:32.484 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.702+0000 7fb58b5d0700 1 -- 172.21.15.17:0/3444418329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 msgr2=0x7fb584105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:32.484 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.702+0000 7fb58b5d0700 1 --2- 172.21.15.17:0/3444418329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584105520 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb5800098a0 tx=0x7fb580009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:32.484 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.703+0000 7fb58b5d0700 1 -- 172.21.15.17:0/3444418329 shutdown_connections 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.703+0000 7fb58b5d0700 1 --2- 172.21.15.17:0/3444418329 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584105520 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.703+0000 7fb58b5d0700 1 -- 172.21.15.17:0/3444418329 >> 172.21.15.17:0/3444418329 conn(0x7fb584100680 msgr2=0x7fb584102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.703+0000 7fb58b5d0700 1 -- 172.21.15.17:0/3444418329 shutdown_connections 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.703+0000 7fb58b5d0700 1 -- 172.21.15.17:0/3444418329 wait complete. 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.703+0000 7fb58b5d0700 1 Processor -- start 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.703+0000 7fb58b5d0700 1 -- start start 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb58b5d0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:32.485 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb58b5d0700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58419a480 con 0x7fb584105100 2023-07-20T02:33:32.486 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb58936c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:32.486 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb58936c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42470/0 (socket says 172.21.15.17:42470) 2023-07-20T02:33:32.486 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb58936c700 1 -- 172.21.15.17:0/2081525211 learned_addr learned my addr 172.21.15.17:0/2081525211 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:32.486 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb58936c700 1 -- 172.21.15.17:0/2081525211 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580009550 con 0x7fb584105100 2023-07-20T02:33:32.486 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb58936c700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584199f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb580009130 tx=0x7fb580011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.704+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb580003ab0 con 0x7fb584105100 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.705+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58419a680 con 0x7fb584105100 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.705+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb580003c10 con 0x7fb584105100 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.705+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5800247d0 con 0x7fb584105100 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.705+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58419aaa0 con 0x7fb584105100 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.705+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb58002d3f0 con 0x7fb584105100 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.706+0000 7fb57a7fc700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb570038710 0x7fb57003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.706+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb58004d860 con 0x7fb584105100 2023-07-20T02:33:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.706+0000 7fb588b6b700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb570038710 0x7fb57003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.706+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb584061da0 con 0x7fb584105100 2023-07-20T02:33:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.706+0000 7fb588b6b700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb570038710 0x7fb57003abd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb574006fd0 tx=0x7fb574006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.711+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb580010970 con 0x7fb584105100 2023-07-20T02:33:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:31.748+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mon.0 v2:172.21.15.17:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb580010970 con 0x7fb584105100 2023-07-20T02:33:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.099+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb584102900 con 0x7fb570038710 2023-07-20T02:33:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.105+0000 7fb57a7fc700 1 -- 172.21.15.17:0/2081525211 <== mgr.14120 v2:172.21.15.17:6800/3625702005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb584102900 con 0x7fb570038710 2023-07-20T02:33:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb570038710 msgr2=0x7fb57003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb570038710 0x7fb57003abd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb574006fd0 tx=0x7fb574006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 msgr2=0x7fb584199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584199f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb580009130 tx=0x7fb580011ad0 comp rx=0 tx=0).stop 2023-07-20T02:33:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 shutdown_connections 2023-07-20T02:33:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fb570038710 0x7fb57003abd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 --2- 172.21.15.17:0/2081525211 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb584105100 0x7fb584199f40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:32.490 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.107+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 >> 172.21.15.17:0/2081525211 conn(0x7fb584100680 msgr2=0x7fb584073a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:32.490 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.108+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 shutdown_connections 2023-07-20T02:33:32.490 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.108+0000 7fb58b5d0700 1 -- 172.21.15.17:0/2081525211 wait complete. 2023-07-20T02:33:32.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:32 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:32.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:32 smithi017 ceph-mon[127345]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:32.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:32 smithi017 ceph-mon[127345]: Saving service mgr spec with placement count:2 2023-07-20T02:33:32.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:32 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:32.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:32 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:33.826 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.959+0000 7fa556be7700 1 Processor -- start 2023-07-20T02:33:33.827 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.959+0000 7fa556be7700 1 -- start start 2023-07-20T02:33:33.827 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.959+0000 7fa556be7700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa5501090d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:33.827 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.959+0000 7fa556be7700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa550079dc0 con 0x7fa550106ce0 2023-07-20T02:33:33.828 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa554983700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa5501090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:33.828 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa554983700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa5501090d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42474/0 (socket says 172.21.15.17:42474) 2023-07-20T02:33:33.828 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa554983700 1 -- 172.21.15.17:0/3299297965 learned_addr learned my addr 172.21.15.17:0/3299297965 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:33.828 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa554983700 1 -- 172.21.15.17:0/3299297965 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa550109610 con 0x7fa550106ce0 2023-07-20T02:33:33.828 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa554983700 1 --2- 172.21.15.17:0/3299297965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa5501090d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e80 tx=0x7fa54800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896252d406d0c5bf server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:33.828 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa54f7fe700 1 -- 172.21.15.17:0/3299297965 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa54800e070 con 0x7fa550106ce0 2023-07-20T02:33:33.829 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa54f7fe700 1 -- 172.21.15.17:0/3299297965 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54800b8c0 con 0x7fa550106ce0 2023-07-20T02:33:33.829 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.960+0000 7fa54f7fe700 1 -- 172.21.15.17:0/3299297965 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5480043e0 con 0x7fa550106ce0 2023-07-20T02:33:33.829 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.961+0000 7fa556be7700 1 -- 172.21.15.17:0/3299297965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 msgr2=0x7fa5501090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:33.829 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.961+0000 7fa556be7700 1 --2- 172.21.15.17:0/3299297965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa5501090d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e80 tx=0x7fa54800b330 comp rx=0 tx=0).stop 2023-07-20T02:33:33.829 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.961+0000 7fa556be7700 1 -- 172.21.15.17:0/3299297965 shutdown_connections 2023-07-20T02:33:33.830 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.961+0000 7fa556be7700 1 --2- 172.21.15.17:0/3299297965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa5501090d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:33.830 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.961+0000 7fa556be7700 1 -- 172.21.15.17:0/3299297965 >> 172.21.15.17:0/3299297965 conn(0x7fa5501006b0 msgr2=0x7fa550102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:33.830 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.961+0000 7fa556be7700 1 -- 172.21.15.17:0/3299297965 shutdown_connections 2023-07-20T02:33:33.832 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.961+0000 7fa556be7700 1 -- 172.21.15.17:0/3299297965 wait complete. 2023-07-20T02:33:33.832 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa556be7700 1 Processor -- start 2023-07-20T02:33:33.832 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa556be7700 1 -- start start 2023-07-20T02:33:33.832 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa556be7700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa550199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:33.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa556be7700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55019a400 con 0x7fa550106ce0 2023-07-20T02:33:33.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa554983700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa550199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:33.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa554983700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa550199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42476/0 (socket says 172.21.15.17:42476) 2023-07-20T02:33:33.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa554983700 1 -- 172.21.15.17:0/3327706134 learned_addr learned my addr 172.21.15.17:0/3327706134 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:33.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa554983700 1 -- 172.21.15.17:0/3327706134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa548009b30 con 0x7fa550106ce0 2023-07-20T02:33:33.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.962+0000 7fa554983700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa550199ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa548009530 tx=0x7fa548003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:33.833 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.963+0000 7fa54dffb700 1 -- 172.21.15.17:0/3327706134 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa54800e040 con 0x7fa550106ce0 2023-07-20T02:33:33.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.963+0000 7fa54dffb700 1 -- 172.21.15.17:0/3327706134 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54800bd70 con 0x7fa550106ce0 2023-07-20T02:33:33.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.963+0000 7fa54dffb700 1 -- 172.21.15.17:0/3327706134 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa54801c6b0 con 0x7fa550106ce0 2023-07-20T02:33:33.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.963+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55019a600 con 0x7fa550106ce0 2023-07-20T02:33:33.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.963+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55019aa80 con 0x7fa550106ce0 2023-07-20T02:33:33.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.964+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa550061da0 con 0x7fa550106ce0 2023-07-20T02:33:33.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.964+0000 7fa54dffb700 1 -- 172.21.15.17:0/3327706134 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa548012a80 con 0x7fa550106ce0 2023-07-20T02:33:33.834 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.964+0000 7fa54dffb700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fa53c038710 0x7fa53c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:33.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.964+0000 7fa54dffb700 1 -- 172.21.15.17:0/3327706134 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa54804c580 con 0x7fa550106ce0 2023-07-20T02:33:33.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.964+0000 7fa54ffff700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fa53c038710 0x7fa53c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:33.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.965+0000 7fa54ffff700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fa53c038710 0x7fa53c03abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa538006fd0 tx=0x7fa538006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:33.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:32.969+0000 7fa54dffb700 1 -- 172.21.15.17:0/3327706134 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa548012430 con 0x7fa550106ce0 2023-07-20T02:33:33.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.330+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa55019d460 con 0x7fa550106ce0 2023-07-20T02:33:33.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.333+0000 7fa54dffb700 1 -- 172.21.15.17:0/3327706134 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa5480170d0 con 0x7fa550106ce0 2023-07-20T02:33:33.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.335+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fa53c038710 msgr2=0x7fa53c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.335+0000 7fa556be7700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fa53c038710 0x7fa53c03abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa538006fd0 tx=0x7fa538006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 msgr2=0x7fa550199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa550199ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa548009530 tx=0x7fa548003d00 comp rx=0 tx=0).stop 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 shutdown_connections 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fa53c038710 0x7fa53c03abd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 --2- 172.21.15.17:0/3327706134 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa550106ce0 0x7fa550199ec0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 >> 172.21.15.17:0/3327706134 conn(0x7fa5501006b0 msgr2=0x7fa550101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:33.836 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 shutdown_connections 2023-07-20T02:33:33.837 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:33.336+0000 7fa556be7700 1 -- 172.21.15.17:0/3327706134 wait complete. 2023-07-20T02:33:34.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:34 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3327706134' entity='client.admin' 2023-07-20T02:33:34.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:34 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:35.071 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.332+0000 7f52d55dd700 1 Processor -- start 2023-07-20T02:33:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.332+0000 7f52d55dd700 1 -- start start 2023-07-20T02:33:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.332+0000 7f52d55dd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d01090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.332+0000 7f52d55dd700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0073f70 con 0x7f52d0106cb0 2023-07-20T02:33:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ceffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d01090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ceffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d01090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42480/0 (socket says 172.21.15.17:42480) 2023-07-20T02:33:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ceffd700 1 -- 172.21.15.17:0/1705283754 learned_addr learned my addr 172.21.15.17:0/1705283754 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ceffd700 1 -- 172.21.15.17:0/1705283754 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d01095e0 con 0x7f52d0106cb0 2023-07-20T02:33:35.073 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ceffd700 1 --2- 172.21.15.17:0/1705283754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d01090a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f52c00098a0 tx=0x7f52c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cad0a90966f090c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:35.073 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ce7fc700 1 -- 172.21.15.17:0/1705283754 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52c0004160 con 0x7f52d0106cb0 2023-07-20T02:33:35.073 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ce7fc700 1 -- 172.21.15.17:0/1705283754 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52c00042c0 con 0x7f52d0106cb0 2023-07-20T02:33:35.073 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.333+0000 7f52ce7fc700 1 -- 172.21.15.17:0/1705283754 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52c00119b0 con 0x7f52d0106cb0 2023-07-20T02:33:35.073 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.334+0000 7f52d55dd700 1 -- 172.21.15.17:0/1705283754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 msgr2=0x7f52d01090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:35.073 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.334+0000 7f52d55dd700 1 --2- 172.21.15.17:0/1705283754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d01090a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f52c00098a0 tx=0x7f52c0009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.334+0000 7f52d55dd700 1 -- 172.21.15.17:0/1705283754 shutdown_connections 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.334+0000 7f52d55dd700 1 --2- 172.21.15.17:0/1705283754 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d01090a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.334+0000 7f52d55dd700 1 -- 172.21.15.17:0/1705283754 >> 172.21.15.17:0/1705283754 conn(0x7f52d0100680 msgr2=0x7f52d0102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.334+0000 7f52d55dd700 1 -- 172.21.15.17:0/1705283754 shutdown_connections 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.334+0000 7f52d55dd700 1 -- 172.21.15.17:0/1705283754 wait complete. 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.335+0000 7f52d55dd700 1 Processor -- start 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.335+0000 7f52d55dd700 1 -- start start 2023-07-20T02:33:35.074 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.335+0000 7f52d55dd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d0199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:35.075 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.335+0000 7f52d55dd700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d019a4a0 con 0x7f52d0106cb0 2023-07-20T02:33:35.075 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ceffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d0199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:35.075 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ceffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d0199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42482/0 (socket says 172.21.15.17:42482) 2023-07-20T02:33:35.075 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ceffd700 1 -- 172.21.15.17:0/1758944934 learned_addr learned my addr 172.21.15.17:0/1758944934 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:35.076 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ceffd700 1 -- 172.21.15.17:0/1758944934 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c0009550 con 0x7f52d0106cb0 2023-07-20T02:33:35.076 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ceffd700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d0199f60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52c0006af0 tx=0x7f52c0011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:35.076 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ccff9700 1 -- 172.21.15.17:0/1758944934 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52c0003a40 con 0x7f52d0106cb0 2023-07-20T02:33:35.076 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ccff9700 1 -- 172.21.15.17:0/1758944934 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52c0003ba0 con 0x7f52d0106cb0 2023-07-20T02:33:35.076 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d019a6a0 con 0x7f52d0106cb0 2023-07-20T02:33:35.076 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52ccff9700 1 -- 172.21.15.17:0/1758944934 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52c00196c0 con 0x7f52d0106cb0 2023-07-20T02:33:35.076 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.336+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d019aac0 con 0x7f52d0106cb0 2023-07-20T02:33:35.077 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.337+0000 7f52ccff9700 1 -- 172.21.15.17:0/1758944934 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f52c0019930 con 0x7f52d0106cb0 2023-07-20T02:33:35.077 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.337+0000 7f52ccff9700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f52bc038710 0x7f52bc03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:35.077 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.337+0000 7f52ccff9700 1 -- 172.21.15.17:0/1758944934 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f52c004ca30 con 0x7f52d0106cb0 2023-07-20T02:33:35.077 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.337+0000 7f52c7fff700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f52bc038710 0x7f52bc03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:35.077 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.338+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d0061da0 con 0x7f52d0106cb0 2023-07-20T02:33:35.077 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.338+0000 7f52c7fff700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f52bc038710 0x7f52bc03abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f52b8006fd0 tx=0x7f52b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:35.077 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.342+0000 7f52ccff9700 1 -- 172.21.15.17:0/1758944934 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f52c0019e60 con 0x7f52d0106cb0 2023-07-20T02:33:35.078 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.713+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f52d019d5f0 con 0x7f52d0106cb0 2023-07-20T02:33:35.078 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.717+0000 7f52ccff9700 1 -- 172.21.15.17:0/1758944934 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f52c0024e40 con 0x7f52d0106cb0 2023-07-20T02:33:35.078 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.719+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f52bc038710 msgr2=0x7f52bc03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:35.078 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.719+0000 7f52d55dd700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f52bc038710 0x7f52bc03abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f52b8006fd0 tx=0x7f52b8006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:35.078 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.719+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 msgr2=0x7f52d0199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:35.078 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.719+0000 7f52d55dd700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d0199f60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52c0006af0 tx=0x7f52c0011a10 comp rx=0 tx=0).stop 2023-07-20T02:33:35.078 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.720+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 shutdown_connections 2023-07-20T02:33:35.079 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.720+0000 7f52d55dd700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f52bc038710 0x7f52bc03abd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:35.079 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.720+0000 7f52d55dd700 1 --2- 172.21.15.17:0/1758944934 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f52d0106cb0 0x7f52d0199f60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:35.079 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.720+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 >> 172.21.15.17:0/1758944934 conn(0x7f52d0100680 msgr2=0x7f52d0101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:35.079 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.720+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 shutdown_connections 2023-07-20T02:33:35.079 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:34.720+0000 7f52d55dd700 1 -- 172.21.15.17:0/1758944934 wait complete. 2023-07-20T02:33:35.079 INFO:teuthology.orchestra.run.smithi017.stdout:Enabling the dashboard module... 2023-07-20T02:33:35.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:35 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1758944934' entity='client.admin' 2023-07-20T02:33:35.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:35 smithi017 ceph-mon[127345]: from='mgr.14120 172.21.15.17:0/573697983' entity='mgr.x' 2023-07-20T02:33:37.017 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:36 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2126109404' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-20T02:33:37.018 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:36 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: ignoring --setuser ceph since I am not root 2023-07-20T02:33:37.018 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:36 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: ignoring --setgroup ceph since I am not root 2023-07-20T02:33:37.018 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:36 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:36.930+0000 7f865be09200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:33:37.018 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:37.028+0000 7f865be09200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:33:37.028 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.552+0000 7f87c9066700 1 Processor -- start 2023-07-20T02:33:37.030 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c9066700 1 -- start start 2023-07-20T02:33:37.030 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c9066700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4106610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:37.030 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c9066700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c4106be0 con 0x7f87c41061f0 2023-07-20T02:33:37.030 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c2d9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:37.030 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c2d9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42486/0 (socket says 172.21.15.17:42486) 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c2d9d700 1 -- 172.21.15.17:0/178717031 learned_addr learned my addr 172.21.15.17:0/178717031 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c2d9d700 1 -- 172.21.15.17:0/178717031 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c41073f0 con 0x7f87c41061f0 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.553+0000 7f87c2d9d700 1 --2- 172.21.15.17:0/178717031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4106610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009e80 tx=0x7f87b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6674ebd7f27d65c7 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c1d9b700 1 -- 172.21.15.17:0/178717031 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87b400e070 con 0x7f87c41061f0 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c1d9b700 1 -- 172.21.15.17:0/178717031 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87b400b8c0 con 0x7f87c41061f0 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c1d9b700 1 -- 172.21.15.17:0/178717031 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87b40043e0 con 0x7f87c41061f0 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c9066700 1 -- 172.21.15.17:0/178717031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 msgr2=0x7f87c4106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c9066700 1 --2- 172.21.15.17:0/178717031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4106610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009e80 tx=0x7f87b400b330 comp rx=0 tx=0).stop 2023-07-20T02:33:37.034 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c9066700 1 -- 172.21.15.17:0/178717031 shutdown_connections 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c9066700 1 --2- 172.21.15.17:0/178717031 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4106610 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c9066700 1 -- 172.21.15.17:0/178717031 >> 172.21.15.17:0/178717031 conn(0x7f87c4101790 msgr2=0x7f87c4103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c9066700 1 -- 172.21.15.17:0/178717031 shutdown_connections 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.554+0000 7f87c9066700 1 -- 172.21.15.17:0/178717031 wait complete. 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.555+0000 7f87c9066700 1 Processor -- start 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.555+0000 7f87c9066700 1 -- start start 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.555+0000 7f87c9066700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.555+0000 7f87c9066700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c419a430 con 0x7f87c41061f0 2023-07-20T02:33:37.036 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.555+0000 7f87c2d9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:37.036 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87c2d9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42488/0 (socket says 172.21.15.17:42488) 2023-07-20T02:33:37.036 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87c2d9d700 1 -- 172.21.15.17:0/2126109404 learned_addr learned my addr 172.21.15.17:0/2126109404 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:37.036 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87c2d9d700 1 -- 172.21.15.17:0/2126109404 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b4009b30 con 0x7f87c41061f0 2023-07-20T02:33:37.036 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87c2d9d700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4199ef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87b4003fb0 tx=0x7f87b4003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:37.036 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87b400e040 con 0x7f87c41061f0 2023-07-20T02:33:37.036 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87b400bd70 con 0x7f87c41061f0 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c419a630 con 0x7f87c41061f0 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87b401c620 con 0x7f87c41061f0 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.556+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c419aab0 con 0x7f87c41061f0 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.557+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f87b401c8e0 con 0x7f87c41061f0 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.557+0000 7f87bbfff700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f87b00386f0 0x7f87b003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.557+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f87b404c540 con 0x7f87c41061f0 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.557+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c4061da0 con 0x7f87c41061f0 2023-07-20T02:33:37.037 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.557+0000 7f87c259c700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f87b00386f0 0x7f87b003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:37.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.558+0000 7f87c259c700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f87b00386f0 0x7f87b003abb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f87ac006fd0 tx=0x7f87ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:37.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:35.562+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f87b4017070 con 0x7f87c41061f0 2023-07-20T02:33:37.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.038+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f87c419d580 con 0x7f87c41061f0 2023-07-20T02:33:37.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.720+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f87b4019d50 con 0x7f87c41061f0 2023-07-20T02:33:37.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.721+0000 7f87bbfff700 1 -- 172.21.15.17:0/2126109404 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f87b4011390 con 0x7f87c41061f0 2023-07-20T02:33:37.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f87b00386f0 msgr2=0x7f87b003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:37.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f87b00386f0 0x7f87b003abb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f87ac006fd0 tx=0x7f87ac006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 msgr2=0x7f87c4199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4199ef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87b4003fb0 tx=0x7f87b4003d00 comp rx=0 tx=0).stop 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 shutdown_connections 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f87b00386f0 0x7f87b003abb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 --2- 172.21.15.17:0/2126109404 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f87c41061f0 0x7f87c4199ef0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.723+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 >> 172.21.15.17:0/2126109404 conn(0x7f87c4101790 msgr2=0x7f87c4102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.724+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 shutdown_connections 2023-07-20T02:33:37.039 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:36.724+0000 7f87c9066700 1 -- 172.21.15.17:0/2126109404 wait complete. 2023-07-20T02:33:37.286 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:37.154+0000 7f865be09200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:33:37.286 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:37.272+0000 7f865be09200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:33:37.693 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:37.486+0000 7f865be09200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:33:37.985 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2126109404' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-20T02:33:37.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-mon[127345]: mgrmap e9: x(active, since 17s) 2023-07-20T02:33:37.986 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:37.885+0000 7f865be09200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:33:38.260 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:37.995+0000 7f865be09200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:33:38.261 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:38 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:38.153+0000 7f865be09200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:33:38.391 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:33:38.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-20T02:33:38.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T02:33:38.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T02:33:38.396 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T02:33:38.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:33:38.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.561+0000 7fe96ab00700 1 Processor -- start 2023-07-20T02:33:38.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.561+0000 7fe96ab00700 1 -- start start 2023-07-20T02:33:38.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.561+0000 7fe96ab00700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964109050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:38.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.561+0000 7fe96ab00700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe964079e40 con 0x7fe964106c60 2023-07-20T02:33:38.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe96889c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964109050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:38.397 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe96889c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964109050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42502/0 (socket says 172.21.15.17:42502) 2023-07-20T02:33:38.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe96889c700 1 -- 172.21.15.17:0/500400640 learned_addr learned my addr 172.21.15.17:0/500400640 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:38.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe96889c700 1 -- 172.21.15.17:0/500400640 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe964109590 con 0x7fe964106c60 2023-07-20T02:33:38.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe96889c700 1 --2- 172.21.15.17:0/500400640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964109050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9580098a0 tx=0x7fe958009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238058ef709ca9cc server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:38.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe9637fe700 1 -- 172.21.15.17:0/500400640 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe958004160 con 0x7fe964106c60 2023-07-20T02:33:38.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe9637fe700 1 -- 172.21.15.17:0/500400640 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9580042c0 con 0x7fe964106c60 2023-07-20T02:33:38.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.562+0000 7fe9637fe700 1 -- 172.21.15.17:0/500400640 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9580119b0 con 0x7fe964106c60 2023-07-20T02:33:38.398 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.563+0000 7fe96ab00700 1 -- 172.21.15.17:0/500400640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 msgr2=0x7fe964109050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:38.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.563+0000 7fe96ab00700 1 --2- 172.21.15.17:0/500400640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964109050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9580098a0 tx=0x7fe958009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:38.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.563+0000 7fe96ab00700 1 -- 172.21.15.17:0/500400640 shutdown_connections 2023-07-20T02:33:38.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.563+0000 7fe96ab00700 1 --2- 172.21.15.17:0/500400640 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964109050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:38.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.563+0000 7fe96ab00700 1 -- 172.21.15.17:0/500400640 >> 172.21.15.17:0/500400640 conn(0x7fe964100670 msgr2=0x7fe964102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:38.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.563+0000 7fe96ab00700 1 -- 172.21.15.17:0/500400640 shutdown_connections 2023-07-20T02:33:38.399 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.563+0000 7fe96ab00700 1 -- 172.21.15.17:0/500400640 wait complete. 2023-07-20T02:33:38.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.564+0000 7fe96ab00700 1 Processor -- start 2023-07-20T02:33:38.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.564+0000 7fe96ab00700 1 -- start start 2023-07-20T02:33:38.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.564+0000 7fe96ab00700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:38.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.564+0000 7fe96ab00700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419a470 con 0x7fe964106c60 2023-07-20T02:33:38.400 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.564+0000 7fe96889c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe96889c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42504/0 (socket says 172.21.15.17:42504) 2023-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe96889c700 1 -- 172.21.15.17:0/76824567 learned_addr learned my addr 172.21.15.17:0/76824567 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe96889c700 1 -- 172.21.15.17:0/76824567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe958009550 con 0x7fe964106c60 2023-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe96889c700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964199f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe958003f00 tx=0x7fe958011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe961ffb700 1 -- 172.21.15.17:0/76824567 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe958003b20 con 0x7fe964106c60 2023-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe961ffb700 1 -- 172.21.15.17:0/76824567 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe958003c80 con 0x7fe964106c60 2023-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96419a670 con 0x7fe964106c60 2023-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe961ffb700 1 -- 172.21.15.17:0/76824567 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe958019770 con 0x7fe964106c60 2023-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.565+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96419aa90 con 0x7fe964106c60 2023-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.566+0000 7fe961ffb700 1 -- 172.21.15.17:0/76824567 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fe958019a30 con 0x7fe964106c60 2023-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.567+0000 7fe961ffb700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 0x7fe95403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.567+0000 7fe961ffb700 1 -- 172.21.15.17:0/76824567 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe95804c9d0 con 0x7fe964106c60 2023-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.567+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe964061da0 con 0x7fe964106c60 2023-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.567+0000 7fe963fff700 1 -- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 msgr2=0x7fe95403ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/3625702005 2023-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.567+0000 7fe963fff700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 0x7fe95403ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.571+0000 7fe961ffb700 1 -- 172.21.15.17:0/76824567 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe958018db0 con 0x7fe964106c60 2023-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.768+0000 7fe963fff700 1 -- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 msgr2=0x7fe95403ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/3625702005 2023-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:37.768+0000 7fe963fff700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 0x7fe95403ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.037+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe96419d5c0 con 0x7fe964106c60 2023-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.037+0000 7fe961ffb700 1 -- 172.21.15.17:0/76824567 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fe95804a070 con 0x7fe964106c60 2023-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 msgr2=0x7fe95403ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 0x7fe95403ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 msgr2=0x7fe964199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964199f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe958003f00 tx=0x7fe958011a30 comp rx=0 tx=0).stop 2023-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 shutdown_connections 2023-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7fe954038760 0x7fe95403ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 --2- 172.21.15.17:0/76824567 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe964106c60 0x7fe964199f30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 >> 172.21.15.17:0/76824567 conn(0x7fe964100670 msgr2=0x7fe964101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 shutdown_connections 2023-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.040+0000 7fe96ab00700 1 -- 172.21.15.17:0/76824567 wait complete. 2023-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for the mgr to restart... 2023-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr epoch 9... 2023-07-20T02:33:38.599 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:38 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:38.270+0000 7f865be09200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:33:38.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:38 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/76824567' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T02:33:38.944 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:38 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:38.801+0000 7f865be09200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:33:39.943 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:39 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:39.490+0000 7f865be09200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:33:39.944 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:39 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:39.683+0000 7f865be09200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:33:40.943 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:40.649+0000 7f865be09200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:33:40.944 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:40.760+0000 7f865be09200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:33:41.321 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:40.981+0000 7f865be09200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:33:41.599 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:41.331+0000 7f865be09200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:33:41.943 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:41.608+0000 7f865be09200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:33:41.944 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:41.791+0000 7f865be09200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:33:42.443 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:42 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:42.124+0000 7f865be09200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:33:43.347 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:42 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:42.953+0000 7f865be09200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:33:43.389 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:43.111+0000 7f865be09200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:33:43.693 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:43.357+0000 7f865be09200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:33:43.694 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:43.494+0000 7f865be09200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:33:44.194 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:43.768+0000 7f865be09200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:33:45.194 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:33:44 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:33:44.809+0000 7f865be09200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:33:45.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: Active manager daemon x restarted 2023-07-20T02:33:45.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: Activating manager daemon x 2023-07-20T02:33:45.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: osdmap e3: 0 total, 0 up, 0 in 2023-07-20T02:33:45.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: mgrmap e10: x(active, starting, since 0.00311426s) 2023-07-20T02:33:45.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:33:45.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T02:33:45.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T02:33:45.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T02:33:45.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T02:33:45.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: Manager daemon x is now available 2023-07-20T02:33:45.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:45.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:45.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T02:33:45.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:45 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T02:33:46.270 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:33:46.271 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-20T02:33:46.271 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T02:33:46.271 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:33:46.271 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.918+0000 7f30834be700 1 Processor -- start 2023-07-20T02:33:46.271 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.918+0000 7f30834be700 1 -- start start 2023-07-20T02:33:46.271 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f30834be700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c1090e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:46.272 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f30834be700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c073f30 con 0x7f307c106cf0 2023-07-20T02:33:46.272 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f308125a700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c1090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:46.272 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f308125a700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c1090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42512/0 (socket says 172.21.15.17:42512) 2023-07-20T02:33:46.272 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f308125a700 1 -- 172.21.15.17:0/2564675638 learned_addr learned my addr 172.21.15.17:0/2564675638 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:46.272 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f308125a700 1 -- 172.21.15.17:0/2564675638 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c109620 con 0x7f307c106cf0 2023-07-20T02:33:46.272 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f308125a700 1 --2- 172.21.15.17:0/2564675638 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c1090e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f30780098a0 tx=0x7f3078009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7757917acec3c137 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:46.272 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.919+0000 7f3073fff700 1 -- 172.21.15.17:0/2564675638 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3078004160 con 0x7f307c106cf0 2023-07-20T02:33:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.920+0000 7f3073fff700 1 -- 172.21.15.17:0/2564675638 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30780042c0 con 0x7f307c106cf0 2023-07-20T02:33:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.920+0000 7f3073fff700 1 -- 172.21.15.17:0/2564675638 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f30780119b0 con 0x7f307c106cf0 2023-07-20T02:33:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.920+0000 7f30834be700 1 -- 172.21.15.17:0/2564675638 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 msgr2=0x7f307c1090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.920+0000 7f30834be700 1 --2- 172.21.15.17:0/2564675638 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c1090e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f30780098a0 tx=0x7f3078009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.920+0000 7f30834be700 1 -- 172.21.15.17:0/2564675638 shutdown_connections 2023-07-20T02:33:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.920+0000 7f30834be700 1 --2- 172.21.15.17:0/2564675638 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c1090e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.920+0000 7f30834be700 1 -- 172.21.15.17:0/2564675638 >> 172.21.15.17:0/2564675638 conn(0x7f307c100680 msgr2=0x7f307c102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:46.274 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.921+0000 7f30834be700 1 -- 172.21.15.17:0/2564675638 shutdown_connections 2023-07-20T02:33:46.274 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.921+0000 7f30834be700 1 -- 172.21.15.17:0/2564675638 wait complete. 2023-07-20T02:33:46.274 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.921+0000 7f30834be700 1 Processor -- start 2023-07-20T02:33:46.274 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.921+0000 7f30834be700 1 -- start start 2023-07-20T02:33:46.274 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f30834be700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:46.274 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f30834be700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c106b90 con 0x7f307c106cf0 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f308125a700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f308125a700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42514/0 (socket says 172.21.15.17:42514) 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f308125a700 1 -- 172.21.15.17:0/2068495982 learned_addr learned my addr 172.21.15.17:0/2068495982 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f308125a700 1 -- 172.21.15.17:0/2068495982 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078009550 con 0x7f307c106cf0 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f308125a700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c106650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3078003f00 tx=0x7f3078011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3078003b20 con 0x7f307c106cf0 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.922+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3078003c80 con 0x7f307c106cf0 2023-07-20T02:33:46.275 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.923+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c104ca0 con 0x7f307c106cf0 2023-07-20T02:33:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.923+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3078019770 con 0x7f307c106cf0 2023-07-20T02:33:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.923+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c1050c0 con 0x7f307c106cf0 2023-07-20T02:33:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.923+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f307801a430 con 0x7f307c106cf0 2023-07-20T02:33:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.924+0000 7f30727fc700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.924+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 --> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f306803b380 con 0x7f30680387b0 2023-07-20T02:33:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.924+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f307804c990 con 0x7f307c106cf0 2023-07-20T02:33:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.924+0000 7f3080a59700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 msgr2=0x7f306803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/3625702005 2023-07-20T02:33:46.277 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:38.924+0000 7f3080a59700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:33:46.277 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:39.125+0000 7f3080a59700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 msgr2=0x7f306803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/3625702005 2023-07-20T02:33:46.277 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:39.125+0000 7f3080a59700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:33:46.277 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:39.526+0000 7f3080a59700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 msgr2=0x7f306803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/3625702005 2023-07-20T02:33:46.277 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:39.526+0000 7f3080a59700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T02:33:46.277 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:40.327+0000 7f3080a59700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 msgr2=0x7f306803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/3625702005 2023-07-20T02:33:46.277 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:40.327+0000 7f3080a59700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:41.928+0000 7f3080a59700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 msgr2=0x7f306803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.17:6800/3625702005 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:41.928+0000 7f3080a59700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:44.824+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f307801ab60 con 0x7f307c106cf0 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:44.824+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 msgr2=0x7f306803ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:44.824+0000 7f30727fc700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.825+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f307804eda0 con 0x7f307c106cf0 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.826+0000 7f30727fc700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f306803bfd0 0x7f306803e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:46.278 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.826+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f306803b380 con 0x7f306803bfd0 2023-07-20T02:33:46.279 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.826+0000 7f3080a59700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f306803bfd0 0x7f306803e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:46.279 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.827+0000 7f3080a59700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f306803bfd0 0x7f306803e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f306c003a10 tx=0x7f306c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:46.279 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.827+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f306803b380 con 0x7f306803bfd0 2023-07-20T02:33:46.279 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.838+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f307c1061f0 con 0x7f306803bfd0 2023-07-20T02:33:46.279 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.838+0000 7f30727fc700 1 -- 172.21.15.17:0/2068495982 <== mgr.14152 v2:172.21.15.17:6800/2503922589 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f307c1061f0 con 0x7f306803bfd0 2023-07-20T02:33:46.279 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f306803bfd0 msgr2=0x7f306803e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:46.279 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f306803bfd0 0x7f306803e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f306c003a10 tx=0x7f306c009270 comp rx=0 tx=0).stop 2023-07-20T02:33:46.280 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 msgr2=0x7f307c106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:46.280 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c106650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3078003f00 tx=0x7f3078011a30 comp rx=0 tx=0).stop 2023-07-20T02:33:46.280 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 shutdown_connections 2023-07-20T02:33:46.280 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f306803bfd0 0x7f306803e3c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:46.280 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f307c106cf0 0x7f307c106650 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:46.280 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 --2- 172.21.15.17:0/2068495982 >> [v2:172.21.15.17:6800/3625702005,v1:172.21.15.17:6801/3625702005] conn(0x7f30680387b0 0x7f306803ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:46.281 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 >> 172.21.15.17:0/2068495982 conn(0x7f307c100680 msgr2=0x7f307c101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:46.281 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 shutdown_connections 2023-07-20T02:33:46.281 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:45.839+0000 7f30834be700 1 -- 172.21.15.17:0/2068495982 wait complete. 2023-07-20T02:33:46.281 INFO:teuthology.orchestra.run.smithi017.stdout:mgr epoch 9 is available 2023-07-20T02:33:46.281 INFO:teuthology.orchestra.run.smithi017.stdout:Generating a dashboard self-signed certificate... 2023-07-20T02:33:47.193 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:46 smithi017 ceph-mon[127345]: mgrmap e11: x(active, since 1.00558s) 2023-07-20T02:33:47.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:46 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:45] ENGINE Bus STARTING 2023-07-20T02:33:47.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:46 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:45] ENGINE Serving on https://172.21.15.17:7150 2023-07-20T02:33:47.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:46 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:46] ENGINE Serving on http://172.21.15.17:8765 2023-07-20T02:33:47.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:46 smithi017 ceph-mon[127345]: [20/Jul/2023:02:33:46] ENGINE Bus STARTED 2023-07-20T02:33:48.119 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-20T02:33:48.120 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.381+0000 7ff36a2a1700 1 Processor -- start 2023-07-20T02:33:48.120 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.381+0000 7ff36a2a1700 1 -- start start 2023-07-20T02:33:48.120 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.381+0000 7ff36a2a1700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff3641090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:48.121 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.381+0000 7ff36a2a1700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364073f70 con 0x7ff364106cb0 2023-07-20T02:33:48.121 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff363fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff3641090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:48.121 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff363fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff3641090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42568/0 (socket says 172.21.15.17:42568) 2023-07-20T02:33:48.121 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff363fff700 1 -- 172.21.15.17:0/3194269788 learned_addr learned my addr 172.21.15.17:0/3194269788 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:48.122 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff363fff700 1 -- 172.21.15.17:0/3194269788 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3641095e0 con 0x7ff364106cb0 2023-07-20T02:33:48.122 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff363fff700 1 --2- 172.21.15.17:0/3194269788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff3641090a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff3540098a0 tx=0x7ff354009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75001eace1bd2544 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:48.122 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff362ffd700 1 -- 172.21.15.17:0/3194269788 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff354004160 con 0x7ff364106cb0 2023-07-20T02:33:48.123 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff362ffd700 1 -- 172.21.15.17:0/3194269788 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3540042c0 con 0x7ff364106cb0 2023-07-20T02:33:48.123 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.382+0000 7ff362ffd700 1 -- 172.21.15.17:0/3194269788 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3540119b0 con 0x7ff364106cb0 2023-07-20T02:33:48.123 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.383+0000 7ff36a2a1700 1 -- 172.21.15.17:0/3194269788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 msgr2=0x7ff3641090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:48.123 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.383+0000 7ff36a2a1700 1 --2- 172.21.15.17:0/3194269788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff3641090a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff3540098a0 tx=0x7ff354009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:48.124 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.383+0000 7ff36a2a1700 1 -- 172.21.15.17:0/3194269788 shutdown_connections 2023-07-20T02:33:48.124 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.383+0000 7ff36a2a1700 1 --2- 172.21.15.17:0/3194269788 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff3641090a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:48.124 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.383+0000 7ff36a2a1700 1 -- 172.21.15.17:0/3194269788 >> 172.21.15.17:0/3194269788 conn(0x7ff364100680 msgr2=0x7ff364102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:48.124 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.383+0000 7ff36a2a1700 1 -- 172.21.15.17:0/3194269788 shutdown_connections 2023-07-20T02:33:48.125 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.383+0000 7ff36a2a1700 1 -- 172.21.15.17:0/3194269788 wait complete. 2023-07-20T02:33:48.125 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.384+0000 7ff36a2a1700 1 Processor -- start 2023-07-20T02:33:48.125 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.384+0000 7ff36a2a1700 1 -- start start 2023-07-20T02:33:48.126 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.384+0000 7ff36a2a1700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff364199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:48.126 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.384+0000 7ff36a2a1700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36419a4a0 con 0x7ff364106cb0 2023-07-20T02:33:48.126 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.384+0000 7ff363fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff364199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:48.126 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.384+0000 7ff363fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff364199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42570/0 (socket says 172.21.15.17:42570) 2023-07-20T02:33:48.126 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.384+0000 7ff363fff700 1 -- 172.21.15.17:0/2207488502 learned_addr learned my addr 172.21.15.17:0/2207488502 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:48.126 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.385+0000 7ff363fff700 1 -- 172.21.15.17:0/2207488502 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354009550 con 0x7ff364106cb0 2023-07-20T02:33:48.127 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.385+0000 7ff363fff700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff364199f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff354006af0 tx=0x7ff354011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:48.127 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.385+0000 7ff3617fa700 1 -- 172.21.15.17:0/2207488502 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff354003a40 con 0x7ff364106cb0 2023-07-20T02:33:48.127 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.385+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36419a6a0 con 0x7ff364106cb0 2023-07-20T02:33:48.127 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.385+0000 7ff3617fa700 1 -- 172.21.15.17:0/2207488502 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff354003ba0 con 0x7ff364106cb0 2023-07-20T02:33:48.127 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.385+0000 7ff3617fa700 1 -- 172.21.15.17:0/2207488502 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3540196c0 con 0x7ff364106cb0 2023-07-20T02:33:48.128 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.385+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36419aac0 con 0x7ff364106cb0 2023-07-20T02:33:48.128 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.386+0000 7ff3617fa700 1 -- 172.21.15.17:0/2207488502 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7ff354019980 con 0x7ff364106cb0 2023-07-20T02:33:48.128 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.386+0000 7ff3617fa700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff34c0385f0 0x7ff34c03aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:48.128 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.386+0000 7ff3617fa700 1 -- 172.21.15.17:0/2207488502 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff35404ca20 con 0x7ff364106cb0 2023-07-20T02:33:48.128 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.386+0000 7ff3637fe700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff34c0385f0 0x7ff34c03aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.386+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff364061da0 con 0x7ff364106cb0 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.387+0000 7ff3637fe700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff34c0385f0 0x7ff34c03aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff358006fd0 tx=0x7ff358006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.391+0000 7ff3617fa700 1 -- 172.21.15.17:0/2207488502 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff3540243f0 con 0x7ff364106cb0 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.764+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ff36419d620 con 0x7ff34c0385f0 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.793+0000 7ff3617fa700 1 -- 172.21.15.17:0/2207488502 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ff36419d620 con 0x7ff34c0385f0 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff34c0385f0 msgr2=0x7ff34c03aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff34c0385f0 0x7ff34c03aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff358006fd0 tx=0x7ff358006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:48.129 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 msgr2=0x7ff364199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:48.130 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff364199f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff354006af0 tx=0x7ff354011a10 comp rx=0 tx=0).stop 2023-07-20T02:33:48.130 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 shutdown_connections 2023-07-20T02:33:48.130 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff34c0385f0 0x7ff34c03aab0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:48.130 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 --2- 172.21.15.17:0/2207488502 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff364106cb0 0x7ff364199f60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:48.130 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 >> 172.21.15.17:0/2207488502 conn(0x7ff364100680 msgr2=0x7ff364101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:48.130 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 shutdown_connections 2023-07-20T02:33:48.130 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:47.795+0000 7ff36a2a1700 1 -- 172.21.15.17:0/2207488502 wait complete. 2023-07-20T02:33:48.131 INFO:teuthology.orchestra.run.smithi017.stdout:Creating initial admin user... 2023-07-20T02:33:48.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:48.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:47 smithi017 ceph-mon[127345]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:48.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:48.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:48.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:47 smithi017 ceph-mon[127345]: mgrmap e12: x(active, since 3s) 2023-07-20T02:33:48.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:49.668 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SdoumfVwjXZ3Z59B7QSVS.b/HuG7EOj8VQq5wFCc8do76cpVFzdIO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689820429, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-20T02:33:49.669 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.573+0000 7f6f42b56700 1 Processor -- start 2023-07-20T02:33:49.669 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.573+0000 7f6f42b56700 1 -- start start 2023-07-20T02:33:49.669 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.573+0000 7f6f42b56700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c104920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:49.669 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.573+0000 7f6f42b56700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c104ef0 con 0x7f6f3c104500 2023-07-20T02:33:49.669 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f408f2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c104920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:49.669 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f408f2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c104920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42574/0 (socket says 172.21.15.17:42574) 2023-07-20T02:33:49.669 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f408f2700 1 -- 172.21.15.17:0/3363413965 learned_addr learned my addr 172.21.15.17:0/3363413965 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:49.670 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f408f2700 1 -- 172.21.15.17:0/3363413965 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c107490 con 0x7f6f3c104500 2023-07-20T02:33:49.670 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f408f2700 1 --2- 172.21.15.17:0/3363413965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c104920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f300098a0 tx=0x7f6f30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c42b2eb4cf9fcc server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:49.670 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f3b7fe700 1 -- 172.21.15.17:0/3363413965 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f30004160 con 0x7f6f3c104500 2023-07-20T02:33:49.670 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f3b7fe700 1 -- 172.21.15.17:0/3363413965 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f300042c0 con 0x7f6f3c104500 2023-07-20T02:33:49.670 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.574+0000 7f6f3b7fe700 1 -- 172.21.15.17:0/3363413965 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f300119b0 con 0x7f6f3c104500 2023-07-20T02:33:49.670 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.575+0000 7f6f42b56700 1 -- 172.21.15.17:0/3363413965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 msgr2=0x7f6f3c104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:49.670 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.575+0000 7f6f42b56700 1 --2- 172.21.15.17:0/3363413965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c104920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f300098a0 tx=0x7f6f30009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.575+0000 7f6f42b56700 1 -- 172.21.15.17:0/3363413965 shutdown_connections 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.575+0000 7f6f42b56700 1 --2- 172.21.15.17:0/3363413965 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c104920 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.575+0000 7f6f42b56700 1 -- 172.21.15.17:0/3363413965 >> 172.21.15.17:0/3363413965 conn(0x7f6f3c0ffe10 msgr2=0x7f6f3c102270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.575+0000 7f6f42b56700 1 -- 172.21.15.17:0/3363413965 shutdown_connections 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.575+0000 7f6f42b56700 1 -- 172.21.15.17:0/3363413965 wait complete. 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.576+0000 7f6f42b56700 1 Processor -- start 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.576+0000 7f6f42b56700 1 -- start start 2023-07-20T02:33:49.671 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.576+0000 7f6f42b56700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:49.672 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.576+0000 7f6f42b56700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c19a4a0 con 0x7f6f3c104500 2023-07-20T02:33:49.672 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.576+0000 7f6f408f2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:49.672 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f408f2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42576/0 (socket says 172.21.15.17:42576) 2023-07-20T02:33:49.672 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f408f2700 1 -- 172.21.15.17:0/1356856075 learned_addr learned my addr 172.21.15.17:0/1356856075 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:49.672 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f408f2700 1 -- 172.21.15.17:0/1356856075 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30009550 con 0x7f6f3c104500 2023-07-20T02:33:49.672 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f408f2700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c199f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6f30003e60 tx=0x7f6f30011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:49.672 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f39ffb700 1 -- 172.21.15.17:0/1356856075 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f30003a60 con 0x7f6f3c104500 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f39ffb700 1 -- 172.21.15.17:0/1356856075 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f30003bc0 con 0x7f6f3c104500 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3c19a6a0 con 0x7f6f3c104500 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.577+0000 7f6f39ffb700 1 -- 172.21.15.17:0/1356856075 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f300196c0 con 0x7f6f3c104500 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.578+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3c19aac0 con 0x7f6f3c104500 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.579+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3c061da0 con 0x7f6f3c104500 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.579+0000 7f6f39ffb700 1 -- 172.21.15.17:0/1356856075 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f6f3001ab60 con 0x7f6f3c104500 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.579+0000 7f6f39ffb700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6f2c038740 0x7f6f2c03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:49.673 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.579+0000 7f6f39ffb700 1 -- 172.21.15.17:0/1356856075 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6f3004caa0 con 0x7f6f3c104500 2023-07-20T02:33:49.674 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.579+0000 7f6f3bfff700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6f2c038740 0x7f6f2c03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:49.674 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.580+0000 7f6f3bfff700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6f2c038740 0x7f6f2c03ac00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f28006fd0 tx=0x7f6f28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:49.674 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.584+0000 7f6f39ffb700 1 -- 172.21.15.17:0/1356856075 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6f30010790 con 0x7f6f3c104500 2023-07-20T02:33:49.674 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:48.976+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6f3c19d810 con 0x7f6f2c038740 2023-07-20T02:33:49.674 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.224+0000 7f6f39ffb700 1 -- 172.21.15.17:0/1356856075 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6f3c19d810 con 0x7f6f2c038740 2023-07-20T02:33:49.674 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6f2c038740 msgr2=0x7f6f2c03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:49.674 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6f2c038740 0x7f6f2c03ac00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f28006fd0 tx=0x7f6f28006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 msgr2=0x7f6f3c199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c199f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6f30003e60 tx=0x7f6f30011a30 comp rx=0 tx=0).stop 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 shutdown_connections 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6f2c038740 0x7f6f2c03ac00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 --2- 172.21.15.17:0/1356856075 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6f3c104500 0x7f6f3c199f60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 >> 172.21.15.17:0/1356856075 conn(0x7f6f3c0ffe10 msgr2=0x7f6f3c1014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 shutdown_connections 2023-07-20T02:33:49.675 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:49.226+0000 7f6f42b56700 1 -- 172.21.15.17:0/1356856075 wait complete. 2023-07-20T02:33:49.676 INFO:teuthology.orchestra.run.smithi017.stdout:Fetching dashboard port number... 2023-07-20T02:33:50.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:50 smithi017 ceph-mon[127345]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:33:50.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:50.915 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 8443 2023-07-20T02:33:50.916 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f035e45e700 1 Processor -- start 2023-07-20T02:33:50.916 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f035e45e700 1 -- start start 2023-07-20T02:33:50.916 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f035e45e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358106630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:50.916 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f035e45e700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0358106c00 con 0x7f0358106210 2023-07-20T02:33:50.916 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f0357fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358106630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:50.916 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f0357fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358106630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42582/0 (socket says 172.21.15.17:42582) 2023-07-20T02:33:50.916 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f0357fff700 1 -- 172.21.15.17:0/2338768714 learned_addr learned my addr 172.21.15.17:0/2338768714 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f0357fff700 1 -- 172.21.15.17:0/2338768714 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0358107410 con 0x7f0358106210 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.152+0000 7f0357fff700 1 --2- 172.21.15.17:0/2338768714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358106630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f03480098a0 tx=0x7f0348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ae0f7c143f4596 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f0356ffd700 1 -- 172.21.15.17:0/2338768714 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0348004160 con 0x7f0358106210 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f0356ffd700 1 -- 172.21.15.17:0/2338768714 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03480042c0 con 0x7f0358106210 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f0356ffd700 1 -- 172.21.15.17:0/2338768714 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f03480119b0 con 0x7f0358106210 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f035e45e700 1 -- 172.21.15.17:0/2338768714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 msgr2=0x7f0358106630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f035e45e700 1 --2- 172.21.15.17:0/2338768714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358106630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f03480098a0 tx=0x7f0348009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:50.917 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f035e45e700 1 -- 172.21.15.17:0/2338768714 shutdown_connections 2023-07-20T02:33:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f035e45e700 1 --2- 172.21.15.17:0/2338768714 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358106630 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f035e45e700 1 -- 172.21.15.17:0/2338768714 >> 172.21.15.17:0/2338768714 conn(0x7f03581017f0 msgr2=0x7f0358103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f035e45e700 1 -- 172.21.15.17:0/2338768714 shutdown_connections 2023-07-20T02:33:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.153+0000 7f035e45e700 1 -- 172.21.15.17:0/2338768714 wait complete. 2023-07-20T02:33:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.154+0000 7f035e45e700 1 Processor -- start 2023-07-20T02:33:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.154+0000 7f035e45e700 1 -- start start 2023-07-20T02:33:50.918 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.154+0000 7f035e45e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.154+0000 7f035e45e700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035819a4a0 con 0x7f0358106210 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.154+0000 7f0357fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.154+0000 7f0357fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42584/0 (socket says 172.21.15.17:42584) 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.154+0000 7f0357fff700 1 -- 172.21.15.17:0/3958568468 learned_addr learned my addr 172.21.15.17:0/3958568468 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.155+0000 7f0357fff700 1 -- 172.21.15.17:0/3958568468 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0348009550 con 0x7f0358106210 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.155+0000 7f0357fff700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358199f60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0348003e60 tx=0x7f0348011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.155+0000 7f03557fa700 1 -- 172.21.15.17:0/3958568468 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0348003a60 con 0x7f0358106210 2023-07-20T02:33:50.919 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.155+0000 7f03557fa700 1 -- 172.21.15.17:0/3958568468 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0348003bc0 con 0x7f0358106210 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.155+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035819a6a0 con 0x7f0358106210 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.155+0000 7f03557fa700 1 -- 172.21.15.17:0/3958568468 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f03480196c0 con 0x7f0358106210 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.155+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035819aac0 con 0x7f0358106210 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.156+0000 7f03557fa700 1 -- 172.21.15.17:0/3958568468 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f0348019930 con 0x7f0358106210 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.156+0000 7f03557fa700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0340038790 0x7f034003ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.156+0000 7f03557fa700 1 -- 172.21.15.17:0/3958568468 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f034804ca00 con 0x7f0358106210 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.156+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0358061da0 con 0x7f0358106210 2023-07-20T02:33:50.920 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.156+0000 7f03577fe700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0340038790 0x7f034003ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:50.921 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.157+0000 7f03577fe700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0340038790 0x7f034003ac50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f034c006fd0 tx=0x7f034c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:50.921 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.161+0000 7f03557fa700 1 -- 172.21.15.17:0/3958568468 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0348019e60 con 0x7f0358106210 2023-07-20T02:33:50.921 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.517+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f035819d5f0 con 0x7f0358106210 2023-07-20T02:33:50.921 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.517+0000 7f03557fa700 1 -- 172.21.15.17:0/3958568468 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0348010b70 con 0x7f0358106210 2023-07-20T02:33:50.921 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0340038790 msgr2=0x7f034003ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:50.921 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0340038790 0x7f034003ac50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f034c006fd0 tx=0x7f034c006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:50.921 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 msgr2=0x7f0358199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358199f60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0348003e60 tx=0x7f0348011a30 comp rx=0 tx=0).stop 2023-07-20T02:33:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 shutdown_connections 2023-07-20T02:33:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0340038790 0x7f034003ac50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 --2- 172.21.15.17:0/3958568468 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0358106210 0x7f0358199f60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 >> 172.21.15.17:0/3958568468 conn(0x7f03581017f0 msgr2=0x7f0358102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 shutdown_connections 2023-07-20T02:33:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:50.520+0000 7f035e45e700 1 -- 172.21.15.17:0/3958568468 wait complete. 2023-07-20T02:33:50.925 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:33:50.925 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2023-07-20T02:33:50.933 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:33:50.933 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2023-07-20T02:33:50.933 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2023-07-20T02:33:50.933 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-20T02:33:50.934 INFO:teuthology.orchestra.run.smithi017.stdout:Ceph Dashboard is now available at: 2023-07-20T02:33:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: URL: https://smithi017:8443/ 2023-07-20T02:33:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: User: admin 2023-07-20T02:33:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: Password: 4cgey3tngn 2023-07-20T02:33:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:50.936 INFO:teuthology.orchestra.run.smithi017.stdout:Saving cluster configuration to /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config directory 2023-07-20T02:33:50.936 INFO:teuthology.orchestra.run.smithi017.stdout:Enabling autotune for osd_memory_target 2023-07-20T02:33:51.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:51 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3958568468' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-20T02:33:52.180 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.421+0000 7f2e5265b700 1 Processor -- start 2023-07-20T02:33:52.184 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.421+0000 7f2e5265b700 1 -- start start 2023-07-20T02:33:52.185 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.421+0000 7f2e5265b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c106610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:52.185 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.421+0000 7f2e5265b700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c106be0 con 0x7f2e4c1061f0 2023-07-20T02:33:52.186 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4bfff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:52.186 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4bfff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42590/0 (socket says 172.21.15.17:42590) 2023-07-20T02:33:52.186 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4bfff700 1 -- 172.21.15.17:0/1425817685 learned_addr learned my addr 172.21.15.17:0/1425817685 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:52.186 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4bfff700 1 -- 172.21.15.17:0/1425817685 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c1073f0 con 0x7f2e4c1061f0 2023-07-20T02:33:52.186 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4bfff700 1 --2- 172.21.15.17:0/1425817685 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c106610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0098a0 tx=0x7f2e3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6596ae99791bff1 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:52.186 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4affd700 1 -- 172.21.15.17:0/1425817685 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e3c004160 con 0x7f2e4c1061f0 2023-07-20T02:33:52.186 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4affd700 1 -- 172.21.15.17:0/1425817685 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e3c0042c0 con 0x7f2e4c1061f0 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.422+0000 7f2e4affd700 1 -- 172.21.15.17:0/1425817685 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e3c0119b0 con 0x7f2e4c1061f0 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.423+0000 7f2e5265b700 1 -- 172.21.15.17:0/1425817685 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 msgr2=0x7f2e4c106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.423+0000 7f2e5265b700 1 --2- 172.21.15.17:0/1425817685 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c106610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0098a0 tx=0x7f2e3c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.423+0000 7f2e5265b700 1 -- 172.21.15.17:0/1425817685 shutdown_connections 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.423+0000 7f2e5265b700 1 --2- 172.21.15.17:0/1425817685 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c106610 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.423+0000 7f2e5265b700 1 -- 172.21.15.17:0/1425817685 >> 172.21.15.17:0/1425817685 conn(0x7f2e4c101770 msgr2=0x7f2e4c103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.423+0000 7f2e5265b700 1 -- 172.21.15.17:0/1425817685 shutdown_connections 2023-07-20T02:33:52.187 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.423+0000 7f2e5265b700 1 -- 172.21.15.17:0/1425817685 wait complete. 2023-07-20T02:33:52.188 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e5265b700 1 Processor -- start 2023-07-20T02:33:52.188 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e5265b700 1 -- start start 2023-07-20T02:33:52.188 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e5265b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c07aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:52.188 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e5265b700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c07b010 con 0x7f2e4c1061f0 2023-07-20T02:33:52.188 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e4bfff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c07aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:52.189 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e4bfff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c07aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42592/0 (socket says 172.21.15.17:42592) 2023-07-20T02:33:52.189 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e4bfff700 1 -- 172.21.15.17:0/3623506902 learned_addr learned my addr 172.21.15.17:0/3623506902 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:52.189 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.424+0000 7f2e4bfff700 1 -- 172.21.15.17:0/3623506902 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c009550 con 0x7f2e4c1061f0 2023-07-20T02:33:52.189 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.425+0000 7f2e4bfff700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c07aad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c003e60 tx=0x7f2e3c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:52.189 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.425+0000 7f2e497fa700 1 -- 172.21.15.17:0/3623506902 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e3c003a60 con 0x7f2e4c1061f0 2023-07-20T02:33:52.189 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.425+0000 7f2e497fa700 1 -- 172.21.15.17:0/3623506902 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e3c003bc0 con 0x7f2e4c1061f0 2023-07-20T02:33:52.189 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.425+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c079120 con 0x7f2e4c1061f0 2023-07-20T02:33:52.190 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.425+0000 7f2e497fa700 1 -- 172.21.15.17:0/3623506902 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e3c0196c0 con 0x7f2e4c1061f0 2023-07-20T02:33:52.190 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.425+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c079540 con 0x7f2e4c1061f0 2023-07-20T02:33:52.190 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.426+0000 7f2e497fa700 1 -- 172.21.15.17:0/3623506902 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f2e3c019930 con 0x7f2e4c1061f0 2023-07-20T02:33:52.190 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.426+0000 7f2e497fa700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f2e38038830 0x7f2e3803acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:52.190 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.426+0000 7f2e497fa700 1 -- 172.21.15.17:0/3623506902 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2e3c04cdb0 con 0x7f2e4c1061f0 2023-07-20T02:33:52.190 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.426+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c10a620 con 0x7f2e4c1061f0 2023-07-20T02:33:52.190 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.426+0000 7f2e4b7fe700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f2e38038830 0x7f2e3803acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:52.191 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.427+0000 7f2e4b7fe700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f2e38038830 0x7f2e3803acf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e34006fd0 tx=0x7f2e34006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:52.191 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.431+0000 7f2e497fa700 1 -- 172.21.15.17:0/3623506902 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2e3c019e60 con 0x7f2e4c1061f0 2023-07-20T02:33:52.191 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.785+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f2e4c10a810 con 0x7f2e4c1061f0 2023-07-20T02:33:52.191 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.785+0000 7f2e497fa700 1 -- 172.21.15.17:0/3623506902 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2e3c010b70 con 0x7f2e4c1061f0 2023-07-20T02:33:52.191 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f2e38038830 msgr2=0x7f2e3803acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:52.191 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f2e38038830 0x7f2e3803acf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e34006fd0 tx=0x7f2e34006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:52.191 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 msgr2=0x7f2e4c07aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:52.192 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c07aad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c003e60 tx=0x7f2e3c011a30 comp rx=0 tx=0).stop 2023-07-20T02:33:52.192 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 shutdown_connections 2023-07-20T02:33:52.192 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f2e38038830 0x7f2e3803acf0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:52.192 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 --2- 172.21.15.17:0/3623506902 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2e4c1061f0 0x7f2e4c07aad0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:52.192 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 >> 172.21.15.17:0/3623506902 conn(0x7f2e4c101770 msgr2=0x7f2e4c102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:52.192 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 shutdown_connections 2023-07-20T02:33:52.192 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:51.788+0000 7f2e5265b700 1 -- 172.21.15.17:0/3623506902 wait complete. 2023-07-20T02:33:52.418 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:52 smithi017 ceph-mon[127345]: mgrmap e13: x(active, since 6s) 2023-07-20T02:33:53.631 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.664+0000 7f3ee41d6700 1 Processor -- start 2023-07-20T02:33:53.631 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.664+0000 7f3ee41d6700 1 -- start start 2023-07-20T02:33:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee41d6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc104920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee41d6700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc104ef0 con 0x7f3edc104500 2023-07-20T02:33:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee1f72700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc104920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee1f72700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc104920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42596/0 (socket says 172.21.15.17:42596) 2023-07-20T02:33:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee1f72700 1 -- 172.21.15.17:0/307869900 learned_addr learned my addr 172.21.15.17:0/307869900 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:53.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee1f72700 1 -- 172.21.15.17:0/307869900 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc107490 con 0x7f3edc104500 2023-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee1f72700 1 --2- 172.21.15.17:0/307869900 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc104920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80098a0 tx=0x7f3ed8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f779a62ec937e692 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee0f70700 1 -- 172.21.15.17:0/307869900 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ed8004160 con 0x7f3edc104500 2023-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee0f70700 1 -- 172.21.15.17:0/307869900 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ed80042c0 con 0x7f3edc104500 2023-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.665+0000 7f3ee0f70700 1 -- 172.21.15.17:0/307869900 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ed80119b0 con 0x7f3edc104500 2023-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.666+0000 7f3ee41d6700 1 -- 172.21.15.17:0/307869900 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 msgr2=0x7f3edc104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.666+0000 7f3ee41d6700 1 --2- 172.21.15.17:0/307869900 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc104920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80098a0 tx=0x7f3ed8009bb0 comp rx=0 tx=0).stop 2023-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.666+0000 7f3ee41d6700 1 -- 172.21.15.17:0/307869900 shutdown_connections 2023-07-20T02:33:53.634 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.666+0000 7f3ee41d6700 1 --2- 172.21.15.17:0/307869900 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc104920 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:53.634 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.666+0000 7f3ee41d6700 1 -- 172.21.15.17:0/307869900 >> 172.21.15.17:0/307869900 conn(0x7f3edc0ffe10 msgr2=0x7f3edc102270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:53.634 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.666+0000 7f3ee41d6700 1 -- 172.21.15.17:0/307869900 shutdown_connections 2023-07-20T02:33:53.634 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.666+0000 7f3ee41d6700 1 -- 172.21.15.17:0/307869900 wait complete. 2023-07-20T02:33:53.634 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee41d6700 1 Processor -- start 2023-07-20T02:33:53.634 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee41d6700 1 -- start start 2023-07-20T02:33:53.634 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee41d6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc106df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee41d6700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc107330 con 0x7f3edc104500 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee1f72700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee1f72700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42598/0 (socket says 172.21.15.17:42598) 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee1f72700 1 -- 172.21.15.17:0/2589062237 learned_addr learned my addr 172.21.15.17:0/2589062237 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.667+0000 7f3ee1f72700 1 -- 172.21.15.17:0/2589062237 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8009550 con 0x7f3edc104500 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.668+0000 7f3ee1f72700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc106df0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8006af0 tx=0x7f3ed8011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.668+0000 7f3eceffd700 1 -- 172.21.15.17:0/2589062237 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ed8003a40 con 0x7f3edc104500 2023-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.668+0000 7f3eceffd700 1 -- 172.21.15.17:0/2589062237 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ed8003ba0 con 0x7f3edc104500 2023-07-20T02:33:53.636 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.668+0000 7f3eceffd700 1 -- 172.21.15.17:0/2589062237 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ed8019690 con 0x7f3edc104500 2023-07-20T02:33:53.636 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.668+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc105440 con 0x7f3edc104500 2023-07-20T02:33:53.636 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.668+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc105860 con 0x7f3edc104500 2023-07-20T02:33:53.636 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.669+0000 7f3eceffd700 1 -- 172.21.15.17:0/2589062237 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3ed801aa60 con 0x7f3edc104500 2023-07-20T02:33:53.636 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.669+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc10a6c0 con 0x7f3edc104500 2023-07-20T02:33:53.636 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.669+0000 7f3eceffd700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3ec80387b0 0x7f3ec803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:53.636 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.669+0000 7f3eceffd700 1 -- 172.21.15.17:0/2589062237 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ed804cb40 con 0x7f3edc104500 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.669+0000 7f3ee1771700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3ec80387b0 0x7f3ec803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.669+0000 7f3ee1771700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3ec80387b0 0x7f3ec803ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006fd0 tx=0x7f3ed0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:52.674+0000 7f3eceffd700 1 -- 172.21.15.17:0/2589062237 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3ed8018710 con 0x7f3edc104500 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.241+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f3edc10a8b0 con 0x7f3edc104500 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.244+0000 7f3eceffd700 1 -- 172.21.15.17:0/2589062237 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f3ed800ba70 con 0x7f3edc104500 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3ec80387b0 msgr2=0x7f3ec803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3ec80387b0 0x7f3ec803ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006fd0 tx=0x7f3ed0006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:53.637 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 msgr2=0x7f3edc106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:53.638 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc106df0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8006af0 tx=0x7f3ed8011a10 comp rx=0 tx=0).stop 2023-07-20T02:33:53.638 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 shutdown_connections 2023-07-20T02:33:53.638 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3ec80387b0 0x7f3ec803ac70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:53.638 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 --2- 172.21.15.17:0/2589062237 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3edc104500 0x7f3edc106df0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:53.638 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 >> 172.21.15.17:0/2589062237 conn(0x7f3edc0ffe10 msgr2=0x7f3edc1014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:53.638 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 shutdown_connections 2023-07-20T02:33:53.638 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr 2023-07-20T02:33:53.246+0000 7f3ee41d6700 1 -- 172.21.15.17:0/2589062237 wait complete. 2023-07-20T02:33:53.639 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-20T02:33:53.639 INFO:teuthology.orchestra.run.smithi017.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-20T02:33:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:33:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.639 INFO:teuthology.orchestra.run.smithi017.stdout:Or, if you are only running a single cluster on this host: 2023-07-20T02:33:53.639 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-20T02:33:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.640 INFO:teuthology.orchestra.run.smithi017.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-20T02:33:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.640 INFO:teuthology.orchestra.run.smithi017.stdout: ceph telemetry on 2023-07-20T02:33:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.641 INFO:teuthology.orchestra.run.smithi017.stdout:For more information see: 2023-07-20T02:33:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-20T02:33:53.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:33:53.641 INFO:teuthology.orchestra.run.smithi017.stdout:Bootstrap complete. 2023-07-20T02:33:53.666 INFO:tasks.cephadm:Fetching config... 2023-07-20T02:33:53.667 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:33:53.667 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-20T02:33:53.684 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-20T02:33:53.685 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:33:53.685 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-20T02:33:53.742 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-20T02:33:53.742 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:33:53.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-20T02:33:53.809 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-20T02:33:53.810 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:33:53.810 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-20T02:33:53.867 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-20T02:33:53.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4PLr4CHxSrpIfvLEMjP/sKT1OFec4vv7Rq5QmHtikMPj5X3pVErPLv5lb5JiUp/EkhA1a1odQXwn32rcwlyidRVBTa3idi2mBejjSkPHZiAamSUFYyoCU6pwgks2Vlwr7Hifeejl9p4tol7PxnKVwa3r9muyNhXdRmjWQx5VMUCQlDG3OWsXjI17iC0qRD02WEzJso9YqUxu8i2xtjQQ51AptEKmx8pVM8a954VI70cwPlM1LVxx+yeD5OuNjLDB62xFxvku8ymiWQL5aY0OdbvzK2/OZenyqJCe4kLzmL6syj2ursYlzz1x3fPtnnykWXp42Y7JVCL6uU9gbICH6Uh1NedX2HUnbe72jW4+hNsDSt0NNmVPiYEtdc3DFQPXqXev3p+De1DYrZs5Nhzci1h1BSDvJD8QgQpb2Y6gAzKNlBc0KhkO4T3uHLXnfYsKCRrtnErsE7YDRvqZbP3cI8oDTNFWZWn7e4+eS59IDRr+OEB6tdHEfc+hNQYAV2oc= ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T02:33:53.947 INFO:teuthology.orchestra.run.smithi017.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4PLr4CHxSrpIfvLEMjP/sKT1OFec4vv7Rq5QmHtikMPj5X3pVErPLv5lb5JiUp/EkhA1a1odQXwn32rcwlyidRVBTa3idi2mBejjSkPHZiAamSUFYyoCU6pwgks2Vlwr7Hifeejl9p4tol7PxnKVwa3r9muyNhXdRmjWQx5VMUCQlDG3OWsXjI17iC0qRD02WEzJso9YqUxu8i2xtjQQ51AptEKmx8pVM8a954VI70cwPlM1LVxx+yeD5OuNjLDB62xFxvku8ymiWQL5aY0OdbvzK2/OZenyqJCe4kLzmL6syj2ursYlzz1x3fPtnnykWXp42Y7JVCL6uU9gbICH6Uh1NedX2HUnbe72jW4+hNsDSt0NNmVPiYEtdc3DFQPXqXev3p+De1DYrZs5Nhzci1h1BSDvJD8QgQpb2Y6gAzKNlBc0KhkO4T3uHLXnfYsKCRrtnErsE7YDRvqZbP3cI8oDTNFWZWn7e4+eS59IDRr+OEB6tdHEfc+hNQYAV2oc= ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:33:53.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4PLr4CHxSrpIfvLEMjP/sKT1OFec4vv7Rq5QmHtikMPj5X3pVErPLv5lb5JiUp/EkhA1a1odQXwn32rcwlyidRVBTa3idi2mBejjSkPHZiAamSUFYyoCU6pwgks2Vlwr7Hifeejl9p4tol7PxnKVwa3r9muyNhXdRmjWQx5VMUCQlDG3OWsXjI17iC0qRD02WEzJso9YqUxu8i2xtjQQ51AptEKmx8pVM8a954VI70cwPlM1LVxx+yeD5OuNjLDB62xFxvku8ymiWQL5aY0OdbvzK2/OZenyqJCe4kLzmL6syj2ursYlzz1x3fPtnnykWXp42Y7JVCL6uU9gbICH6Uh1NedX2HUnbe72jW4+hNsDSt0NNmVPiYEtdc3DFQPXqXev3p+De1DYrZs5Nhzci1h1BSDvJD8QgQpb2Y6gAzKNlBc0KhkO4T3uHLXnfYsKCRrtnErsE7YDRvqZbP3cI8oDTNFWZWn7e4+eS59IDRr+OEB6tdHEfc+hNQYAV2oc= ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T02:33:54.006 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4PLr4CHxSrpIfvLEMjP/sKT1OFec4vv7Rq5QmHtikMPj5X3pVErPLv5lb5JiUp/EkhA1a1odQXwn32rcwlyidRVBTa3idi2mBejjSkPHZiAamSUFYyoCU6pwgks2Vlwr7Hifeejl9p4tol7PxnKVwa3r9muyNhXdRmjWQx5VMUCQlDG3OWsXjI17iC0qRD02WEzJso9YqUxu8i2xtjQQ51AptEKmx8pVM8a954VI70cwPlM1LVxx+yeD5OuNjLDB62xFxvku8ymiWQL5aY0OdbvzK2/OZenyqJCe4kLzmL6syj2ursYlzz1x3fPtnnykWXp42Y7JVCL6uU9gbICH6Uh1NedX2HUnbe72jW4+hNsDSt0NNmVPiYEtdc3DFQPXqXev3p+De1DYrZs5Nhzci1h1BSDvJD8QgQpb2Y6gAzKNlBc0KhkO4T3uHLXnfYsKCRrtnErsE7YDRvqZbP3cI8oDTNFWZWn7e4+eS59IDRr+OEB6tdHEfc+hNQYAV2oc= ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:33:54.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4PLr4CHxSrpIfvLEMjP/sKT1OFec4vv7Rq5QmHtikMPj5X3pVErPLv5lb5JiUp/EkhA1a1odQXwn32rcwlyidRVBTa3idi2mBejjSkPHZiAamSUFYyoCU6pwgks2Vlwr7Hifeejl9p4tol7PxnKVwa3r9muyNhXdRmjWQx5VMUCQlDG3OWsXjI17iC0qRD02WEzJso9YqUxu8i2xtjQQ51AptEKmx8pVM8a954VI70cwPlM1LVxx+yeD5OuNjLDB62xFxvku8ymiWQL5aY0OdbvzK2/OZenyqJCe4kLzmL6syj2ursYlzz1x3fPtnnykWXp42Y7JVCL6uU9gbICH6Uh1NedX2HUnbe72jW4+hNsDSt0NNmVPiYEtdc3DFQPXqXev3p+De1DYrZs5Nhzci1h1BSDvJD8QgQpb2Y6gAzKNlBc0KhkO4T3uHLXnfYsKCRrtnErsE7YDRvqZbP3cI8oDTNFWZWn7e4+eS59IDRr+OEB6tdHEfc+hNQYAV2oc= ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T02:33:54.064 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4PLr4CHxSrpIfvLEMjP/sKT1OFec4vv7Rq5QmHtikMPj5X3pVErPLv5lb5JiUp/EkhA1a1odQXwn32rcwlyidRVBTa3idi2mBejjSkPHZiAamSUFYyoCU6pwgks2Vlwr7Hifeejl9p4tol7PxnKVwa3r9muyNhXdRmjWQx5VMUCQlDG3OWsXjI17iC0qRD02WEzJso9YqUxu8i2xtjQQ51AptEKmx8pVM8a954VI70cwPlM1LVxx+yeD5OuNjLDB62xFxvku8ymiWQL5aY0OdbvzK2/OZenyqJCe4kLzmL6syj2ursYlzz1x3fPtnnykWXp42Y7JVCL6uU9gbICH6Uh1NedX2HUnbe72jW4+hNsDSt0NNmVPiYEtdc3DFQPXqXev3p+De1DYrZs5Nhzci1h1BSDvJD8QgQpb2Y6gAzKNlBc0KhkO4T3uHLXnfYsKCRrtnErsE7YDRvqZbP3cI8oDTNFWZWn7e4+eS59IDRr+OEB6tdHEfc+hNQYAV2oc= ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:33:54.077 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-20T02:33:54.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:54 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2589062237' entity='client.admin' 2023-07-20T02:33:54.645 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:33:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.494+0000 7f704891e700 1 -- 172.21.15.17:0/3124964482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 msgr2=0x7f7044100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:56.486 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.494+0000 7f704891e700 1 --2- 172.21.15.17:0/3124964482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f7044100c10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7034009910 tx=0x7f7034009c20 comp rx=0 tx=0).stop 2023-07-20T02:33:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.495+0000 7f704891e700 1 -- 172.21.15.17:0/3124964482 shutdown_connections 2023-07-20T02:33:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.495+0000 7f704891e700 1 --2- 172.21.15.17:0/3124964482 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f7044100c10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.495+0000 7f704891e700 1 -- 172.21.15.17:0/3124964482 >> 172.21.15.17:0/3124964482 conn(0x7f70440f9e70 msgr2=0x7f70440fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.495+0000 7f704891e700 1 -- 172.21.15.17:0/3124964482 shutdown_connections 2023-07-20T02:33:56.487 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.495+0000 7f704891e700 1 -- 172.21.15.17:0/3124964482 wait complete. 2023-07-20T02:33:56.488 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.495+0000 7f704891e700 1 Processor -- start 2023-07-20T02:33:56.488 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.495+0000 7f704891e700 1 -- start start 2023-07-20T02:33:56.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704891e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f70440fee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:56.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704891e700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7044198080 con 0x7f70441007f0 2023-07-20T02:33:56.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704259c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f70440fee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:56.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704259c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f70440fee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42604/0 (socket says 172.21.15.17:42604) 2023-07-20T02:33:56.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704259c700 1 -- 172.21.15.17:0/4156594791 learned_addr learned my addr 172.21.15.17:0/4156594791 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:56.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704259c700 1 -- 172.21.15.17:0/4156594791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70340095f0 con 0x7f70441007f0 2023-07-20T02:33:56.489 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704259c700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f70440fee30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7034004f40 tx=0x7f70340057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f703b7fe700 1 -- 172.21.15.17:0/4156594791 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f703401d070 con 0x7f70441007f0 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f703b7fe700 1 -- 172.21.15.17:0/4156594791 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70340044e0 con 0x7f70441007f0 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70441981c0 con 0x7f70441007f0 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f703b7fe700 1 -- 172.21.15.17:0/4156594791 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7034003a20 con 0x7f70441007f0 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.496+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7044198600 con 0x7f70441007f0 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.497+0000 7f703b7fe700 1 -- 172.21.15.17:0/4156594791 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f7034003c50 con 0x7f70441007f0 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.497+0000 7f703b7fe700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7030038800 0x7f703003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:56.490 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.497+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704418dd30 con 0x7f70441007f0 2023-07-20T02:33:56.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.497+0000 7f703b7fe700 1 -- 172.21.15.17:0/4156594791 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f703404cb70 con 0x7f70441007f0 2023-07-20T02:33:56.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.498+0000 7f7041d9b700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7030038800 0x7f703003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:56.491 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.498+0000 7f7041d9b700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7030038800 0x7f703003acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f702c006fd0 tx=0x7f702c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:56.493 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.502+0000 7f703b7fe700 1 -- 172.21.15.17:0/4156594791 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f703404ee60 con 0x7f70441007f0 2023-07-20T02:33:56.688 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.696+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f704402c970 con 0x7f70441007f0 2023-07-20T02:33:56.691 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.700+0000 7f703b7fe700 1 -- 172.21.15.17:0/4156594791 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f703402a540 con 0x7f70441007f0 2023-07-20T02:33:56.693 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7030038800 msgr2=0x7f703003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:56.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7030038800 0x7f703003acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f702c006fd0 tx=0x7f702c006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:56.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 msgr2=0x7f70440fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:56.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f70440fee30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7034004f40 tx=0x7f70340057c0 comp rx=0 tx=0).stop 2023-07-20T02:33:56.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 shutdown_connections 2023-07-20T02:33:56.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7030038800 0x7f703003acc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:56.695 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 --2- 172.21.15.17:0/4156594791 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f70441007f0 0x7f70440fee30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:56.695 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 >> 172.21.15.17:0/4156594791 conn(0x7f70440f9e70 msgr2=0x7f70440fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:56.695 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.702+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 shutdown_connections 2023-07-20T02:33:56.695 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:56.703+0000 7f704891e700 1 -- 172.21.15.17:0/4156594791 wait complete. 2023-07-20T02:33:57.077 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-20T02:33:57.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-20T02:33:57.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:57.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:57.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:33:57.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:57.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:57.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:57.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4156594791' entity='client.admin' 2023-07-20T02:33:57.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:33:57.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:33:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:33:57.635 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:33:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.805+0000 7f432760b700 1 -- 172.21.15.17:0/2915433039 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 msgr2=0x7f4320103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.805+0000 7f432760b700 1 --2- 172.21.15.17:0/2915433039 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f4320103160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4310009910 tx=0x7f4310009c20 comp rx=0 tx=0).stop 2023-07-20T02:33:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.805+0000 7f432760b700 1 -- 172.21.15.17:0/2915433039 shutdown_connections 2023-07-20T02:33:58.798 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.805+0000 7f432760b700 1 --2- 172.21.15.17:0/2915433039 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f4320103160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.805+0000 7f432760b700 1 -- 172.21.15.17:0/2915433039 >> 172.21.15.17:0/2915433039 conn(0x7f43200fa700 msgr2=0x7f43200fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.805+0000 7f432760b700 1 -- 172.21.15.17:0/2915433039 shutdown_connections 2023-07-20T02:33:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.805+0000 7f432760b700 1 -- 172.21.15.17:0/2915433039 wait complete. 2023-07-20T02:33:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f432760b700 1 Processor -- start 2023-07-20T02:33:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f432760b700 1 -- start start 2023-07-20T02:33:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f432760b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f43201006d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:58.799 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f432760b700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4320100c10 con 0x7f4320100d70 2023-07-20T02:33:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f43253a7700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f43201006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f43253a7700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f43201006d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42610/0 (socket says 172.21.15.17:42610) 2023-07-20T02:33:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f43253a7700 1 -- 172.21.15.17:0/1464486608 learned_addr learned my addr 172.21.15.17:0/1464486608 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:33:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.806+0000 7f43253a7700 1 -- 172.21.15.17:0/1464486608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43100095f0 con 0x7f4320100d70 2023-07-20T02:33:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.807+0000 7f43253a7700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f43201006d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4310004f40 tx=0x7f43100057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:58.800 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.807+0000 7f43167fc700 1 -- 172.21.15.17:0/1464486608 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431001d070 con 0x7f4320100d70 2023-07-20T02:33:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.807+0000 7f43167fc700 1 -- 172.21.15.17:0/1464486608 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43100044e0 con 0x7f4320100d70 2023-07-20T02:33:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.807+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43200fed20 con 0x7f4320100d70 2023-07-20T02:33:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.807+0000 7f43167fc700 1 -- 172.21.15.17:0/1464486608 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4310003a20 con 0x7f4320100d70 2023-07-20T02:33:58.801 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.807+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43200ff160 con 0x7f4320100d70 2023-07-20T02:33:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.808+0000 7f43167fc700 1 -- 172.21.15.17:0/1464486608 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4310003c50 con 0x7f4320100d70 2023-07-20T02:33:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.808+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4320061da0 con 0x7f4320100d70 2023-07-20T02:33:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.808+0000 7f43167fc700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f430c038800 0x7f430c03acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:33:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.808+0000 7f43167fc700 1 -- 172.21.15.17:0/1464486608 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f431004ca00 con 0x7f4320100d70 2023-07-20T02:33:58.802 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.808+0000 7f4324ba6700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f430c038800 0x7f430c03acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:33:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.808+0000 7f4324ba6700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f430c038800 0x7f430c03acc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f431c006fd0 tx=0x7f431c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:33:58.803 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:58.812+0000 7f43167fc700 1 -- 172.21.15.17:0/1464486608 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f431002a540 con 0x7f4320100d70 2023-07-20T02:33:59.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.016+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f43200ff400 con 0x7f430c038800 2023-07-20T02:33:59.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.019+0000 7f43167fc700 1 -- 172.21.15.17:0/1464486608 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f43200ff400 con 0x7f430c038800 2023-07-20T02:33:59.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f430c038800 msgr2=0x7f430c03acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:59.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f430c038800 0x7f430c03acc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f431c006fd0 tx=0x7f431c006e40 comp rx=0 tx=0).stop 2023-07-20T02:33:59.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 msgr2=0x7f43201006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:33:59.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f43201006d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4310004f40 tx=0x7f43100057c0 comp rx=0 tx=0).stop 2023-07-20T02:33:59.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 shutdown_connections 2023-07-20T02:33:59.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f430c038800 0x7f430c03acc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:59.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 --2- 172.21.15.17:0/1464486608 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4320100d70 0x7f43201006d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:33:59.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 >> 172.21.15.17:0/1464486608 conn(0x7f43200fa700 msgr2=0x7f43200fb440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:33:59.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 shutdown_connections 2023-07-20T02:33:59.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:33:59.022+0000 7f432760b700 1 -- 172.21.15.17:0/1464486608 wait complete. 2023-07-20T02:33:59.481 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2023-07-20T02:33:59.482 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:33:59.483 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2023-07-20T02:33:59.503 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:33:59.504 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:33:59.562 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2023-07-20T02:33:59.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch host add smithi146 2023-07-20T02:34:00.137 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:00.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:00.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:00.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:00.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:00.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:00.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: Updating smithi017:/etc/ceph/ceph.conf 2023-07-20T02:34:00.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: Updating smithi017:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:00.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:00 smithi017 ceph-mon[127345]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:01.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.232+0000 7f18a807a700 1 -- 172.21.15.17:0/3221198507 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 msgr2=0x7f18a0100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:01.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.232+0000 7f18a807a700 1 --2- 172.21.15.17:0/3221198507 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a0100830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1890009910 tx=0x7f1890009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:01.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.232+0000 7f18a807a700 1 -- 172.21.15.17:0/3221198507 shutdown_connections 2023-07-20T02:34:01.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.232+0000 7f18a807a700 1 --2- 172.21.15.17:0/3221198507 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a0100830 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:01.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.232+0000 7f18a807a700 1 -- 172.21.15.17:0/3221198507 >> 172.21.15.17:0/3221198507 conn(0x7f18a00fbd00 msgr2=0x7f18a00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:01.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.232+0000 7f18a807a700 1 -- 172.21.15.17:0/3221198507 shutdown_connections 2023-07-20T02:34:01.225 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.232+0000 7f18a807a700 1 -- 172.21.15.17:0/3221198507 wait complete. 2023-07-20T02:34:01.226 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.233+0000 7f18a807a700 1 Processor -- start 2023-07-20T02:34:01.226 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.233+0000 7f18a807a700 1 -- start start 2023-07-20T02:34:01.226 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.233+0000 7f18a807a700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a01956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:01.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.233+0000 7f18a807a700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0195c00 con 0x7f18a0100450 2023-07-20T02:34:01.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.233+0000 7f18a5e16700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a01956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:01.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.233+0000 7f18a5e16700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a01956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42616/0 (socket says 172.21.15.17:42616) 2023-07-20T02:34:01.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.233+0000 7f18a5e16700 1 -- 172.21.15.17:0/959190196 learned_addr learned my addr 172.21.15.17:0/959190196 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:34:01.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.234+0000 7f18a5e16700 1 -- 172.21.15.17:0/959190196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18900095f0 con 0x7f18a0100450 2023-07-20T02:34:01.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.234+0000 7f18a5e16700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a01956c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1890006010 tx=0x7f1890004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:01.227 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.234+0000 7f1896ffd700 1 -- 172.21.15.17:0/959190196 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f189001d070 con 0x7f18a0100450 2023-07-20T02:34:01.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.234+0000 7f1896ffd700 1 -- 172.21.15.17:0/959190196 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f189000bc10 con 0x7f18a0100450 2023-07-20T02:34:01.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.234+0000 7f1896ffd700 1 -- 172.21.15.17:0/959190196 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1890003910 con 0x7f18a0100450 2023-07-20T02:34:01.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.234+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a0195e00 con 0x7f18a0100450 2023-07-20T02:34:01.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.234+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a0196220 con 0x7f18a0100450 2023-07-20T02:34:01.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.235+0000 7f1896ffd700 1 -- 172.21.15.17:0/959190196 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f189000bd80 con 0x7f18a0100450 2023-07-20T02:34:01.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.235+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a0061da0 con 0x7f18a0100450 2023-07-20T02:34:01.228 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.235+0000 7f1896ffd700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f188c038800 0x7f188c03acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:01.229 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.235+0000 7f1896ffd700 1 -- 172.21.15.17:0/959190196 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f189004c4a0 con 0x7f18a0100450 2023-07-20T02:34:01.229 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.235+0000 7f18a5615700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f188c038800 0x7f188c03acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:01.229 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.236+0000 7f18a5615700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f188c038800 0x7f188c03acc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:01.231 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.240+0000 7f1896ffd700 1 -- 172.21.15.17:0/959190196 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1890029360 con 0x7f18a0100450 2023-07-20T02:34:01.432 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:01.441+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}) v1 -- 0x7f18a0108210 con 0x7f188c038800 2023-07-20T02:34:01.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:01 smithi017 ceph-mon[127345]: Updating smithi017:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:34:01.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:01 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:01.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:01 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:01.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:01 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:02.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:02 smithi017 ceph-mon[127345]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:03.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:03 smithi017 ceph-mon[127345]: Deploying cephadm binary to smithi146 2023-07-20T02:34:04.239 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.247+0000 7f1896ffd700 1 -- 172.21.15.17:0/959190196 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f18a0108210 con 0x7f188c038800 2023-07-20T02:34:04.239 INFO:teuthology.orchestra.run.smithi017.stdout:Added host 'smithi146' with addr '172.21.15.146' 2023-07-20T02:34:04.241 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f188c038800 msgr2=0x7f188c03acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:04.241 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f188c038800 0x7f188c03acc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).stop 2023-07-20T02:34:04.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 msgr2=0x7f18a01956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:04.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a01956c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1890006010 tx=0x7f1890004ff0 comp rx=0 tx=0).stop 2023-07-20T02:34:04.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 shutdown_connections 2023-07-20T02:34:04.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f188c038800 0x7f188c03acc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:04.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 --2- 172.21.15.17:0/959190196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f18a0100450 0x7f18a01956c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:04.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 >> 172.21.15.17:0/959190196 conn(0x7f18a00fbd00 msgr2=0x7f18a01046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:04.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 shutdown_connections 2023-07-20T02:34:04.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:04.250+0000 7f18a807a700 1 -- 172.21.15.17:0/959190196 wait complete. 2023-07-20T02:34:04.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T02:34:05.169 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:05.573 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:05.573 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:05 smithi017 ceph-mon[127345]: Added host smithi146 2023-07-20T02:34:05.573 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:05.574 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:06.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.355+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1851473627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 msgr2=0x7f98a8101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:06.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.355+0000 7f98ae7ad700 1 --2- 172.21.15.17:0/1851473627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8101780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9898009910 tx=0x7f9898009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:06.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.355+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1851473627 shutdown_connections 2023-07-20T02:34:06.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.355+0000 7f98ae7ad700 1 --2- 172.21.15.17:0/1851473627 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8101780 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:06.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.355+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1851473627 >> 172.21.15.17:0/1851473627 conn(0x7f98a80fabf0 msgr2=0x7f98a80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:06.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.355+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1851473627 shutdown_connections 2023-07-20T02:34:06.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.355+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1851473627 wait complete. 2023-07-20T02:34:06.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.356+0000 7f98ae7ad700 1 Processor -- start 2023-07-20T02:34:06.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.356+0000 7f98ae7ad700 1 -- start start 2023-07-20T02:34:06.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.356+0000 7f98ae7ad700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8195710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:06.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.356+0000 7f98ae7ad700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a8195c50 con 0x7f98a81013a0 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.356+0000 7f98a7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8195710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.356+0000 7f98a7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8195710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42642/0 (socket says 172.21.15.17:42642) 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.356+0000 7f98a7fff700 1 -- 172.21.15.17:0/1999972655 learned_addr learned my addr 172.21.15.17:0/1999972655 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.357+0000 7f98a7fff700 1 -- 172.21.15.17:0/1999972655 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980095f0 con 0x7f98a81013a0 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.357+0000 7f98a7fff700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8195710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9898005850 tx=0x7f9898004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.357+0000 7f98a57fa700 1 -- 172.21.15.17:0/1999972655 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f989801d070 con 0x7f98a81013a0 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.357+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a8195e50 con 0x7f98a81013a0 2023-07-20T02:34:06.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.357+0000 7f98a57fa700 1 -- 172.21.15.17:0/1999972655 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9898004370 con 0x7f98a81013a0 2023-07-20T02:34:06.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.357+0000 7f98a57fa700 1 -- 172.21.15.17:0/1999972655 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9898003ab0 con 0x7f98a81013a0 2023-07-20T02:34:06.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.357+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a8196270 con 0x7f98a81013a0 2023-07-20T02:34:06.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.358+0000 7f98a57fa700 1 -- 172.21.15.17:0/1999972655 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f989800f460 con 0x7f98a81013a0 2023-07-20T02:34:06.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.358+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a8061da0 con 0x7f98a81013a0 2023-07-20T02:34:06.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.358+0000 7f98a57fa700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f98940387b0 0x7f989403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:06.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.358+0000 7f98a57fa700 1 -- 172.21.15.17:0/1999972655 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f989804c5e0 con 0x7f98a81013a0 2023-07-20T02:34:06.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.358+0000 7f98a77fe700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f98940387b0 0x7f989403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:06.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.358+0000 7f98a77fe700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f98940387b0 0x7f989403ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:06.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.363+0000 7f98a57fa700 1 -- 172.21.15.17:0/1999972655 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9898029950 con 0x7f98a81013a0 2023-07-20T02:34:06.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:06 smithi017 ceph-mon[127345]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:06.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:06.557 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.565+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98a8107360 con 0x7f98940387b0 2023-07-20T02:34:06.558 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.566+0000 7f98a57fa700 1 -- 172.21.15.17:0/1999972655 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f98a8107360 con 0x7f98940387b0 2023-07-20T02:34:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:34:06.559 INFO:teuthology.orchestra.run.smithi017.stdout:[{"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}] 2023-07-20T02:34:06.560 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.568+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f98940387b0 msgr2=0x7f989403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:06.560 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.568+0000 7f98ae7ad700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f98940387b0 0x7f989403ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890006e40 comp rx=0 tx=0).stop 2023-07-20T02:34:06.560 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.568+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 msgr2=0x7f98a8195710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:06.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.568+0000 7f98ae7ad700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8195710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9898005850 tx=0x7f9898004ff0 comp rx=0 tx=0).stop 2023-07-20T02:34:06.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.569+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 shutdown_connections 2023-07-20T02:34:06.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.569+0000 7f98ae7ad700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f98940387b0 0x7f989403ac70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:06.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.569+0000 7f98ae7ad700 1 --2- 172.21.15.17:0/1999972655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f98a81013a0 0x7f98a8195710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:06.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.569+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 >> 172.21.15.17:0/1999972655 conn(0x7f98a80fabf0 msgr2=0x7f98a81049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:06.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.569+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 shutdown_connections 2023-07-20T02:34:06.561 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:06.569+0000 7f98ae7ad700 1 -- 172.21.15.17:0/1999972655 wait complete. 2023-07-20T02:34:06.926 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi151 2023-07-20T02:34:06.927 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:34:06.927 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.conf 2023-07-20T02:34:06.947 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:34:06.947 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:07.007 INFO:tasks.cephadm:Adding host smithi151 to orchestrator... 2023-07-20T02:34:07.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch host add smithi151 2023-07-20T02:34:07.564 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:07.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:07 smithi017 ceph-mon[127345]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:34:07.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:07 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:08.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.772+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/2539831356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 msgr2=0x7f0b38105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:08.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.772+0000 7f0b3ceb6700 1 --2- 172.21.15.17:0/2539831356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b38105560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009910 tx=0x7f0b28009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:08.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.773+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/2539831356 shutdown_connections 2023-07-20T02:34:08.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.773+0000 7f0b3ceb6700 1 --2- 172.21.15.17:0/2539831356 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b38105560 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:08.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.773+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/2539831356 >> 172.21.15.17:0/2539831356 conn(0x7f0b38074c70 msgr2=0x7f0b38075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:08.765 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.773+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/2539831356 shutdown_connections 2023-07-20T02:34:08.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.773+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/2539831356 wait complete. 2023-07-20T02:34:08.766 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3ceb6700 1 Processor -- start 2023-07-20T02:34:08.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3ceb6700 1 -- start start 2023-07-20T02:34:08.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3ceb6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b381956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:08.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3ceb6700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38195bf0 con 0x7f0b38105180 2023-07-20T02:34:08.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3659c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b381956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:08.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3659c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b381956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42648/0 (socket says 172.21.15.17:42648) 2023-07-20T02:34:08.767 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3659c700 1 -- 172.21.15.17:0/1679742136 learned_addr learned my addr 172.21.15.17:0/1679742136 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:34:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.774+0000 7f0b3659c700 1 -- 172.21.15.17:0/1679742136 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b280095f0 con 0x7f0b38105180 2023-07-20T02:34:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.775+0000 7f0b3659c700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b381956b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0b28006010 tx=0x7f0b28004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.775+0000 7f0b277fe700 1 -- 172.21.15.17:0/1679742136 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b2801d070 con 0x7f0b38105180 2023-07-20T02:34:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.775+0000 7f0b277fe700 1 -- 172.21.15.17:0/1679742136 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b2800bbb0 con 0x7f0b38105180 2023-07-20T02:34:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.775+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b38195e50 con 0x7f0b38105180 2023-07-20T02:34:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.775+0000 7f0b277fe700 1 -- 172.21.15.17:0/1679742136 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b28003930 con 0x7f0b38105180 2023-07-20T02:34:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.775+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b381962d0 con 0x7f0b38105180 2023-07-20T02:34:08.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.776+0000 7f0b277fe700 1 -- 172.21.15.17:0/1679742136 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f0b28003b70 con 0x7f0b38105180 2023-07-20T02:34:08.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.776+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b38061da0 con 0x7f0b38105180 2023-07-20T02:34:08.769 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.776+0000 7f0b277fe700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0b200387b0 0x7f0b2003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:08.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.776+0000 7f0b277fe700 1 -- 172.21.15.17:0/1679742136 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0b2804c6f0 con 0x7f0b38105180 2023-07-20T02:34:08.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.776+0000 7f0b35d9b700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0b200387b0 0x7f0b2003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:08.770 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.777+0000 7f0b35d9b700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0b200387b0 0x7f0b2003ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c006fd0 tx=0x7f0b2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:08.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.781+0000 7f0b277fe700 1 -- 172.21.15.17:0/1679742136 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0b28029950 con 0x7f0b38105180 2023-07-20T02:34:08.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:08 smithi017 ceph-mon[127345]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:08.976 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:08.984+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b38104450 con 0x7f0b200387b0 2023-07-20T02:34:10.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:10.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:10.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:10.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:10.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:10.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:10.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:34:10.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:10.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:10.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: Updating smithi146:/etc/ceph/ceph.conf 2023-07-20T02:34:10.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: Deploying cephadm binary to smithi151 2023-07-20T02:34:10.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:10 smithi017 ceph-mon[127345]: Updating smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:11.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:11 smithi017 ceph-mon[127345]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:11.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.820+0000 7f0b277fe700 1 -- 172.21.15.17:0/1679742136 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0b38104450 con 0x7f0b200387b0 2023-07-20T02:34:11.813 INFO:teuthology.orchestra.run.smithi017.stdout:Added host 'smithi151' with addr '172.21.15.151' 2023-07-20T02:34:11.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0b200387b0 msgr2=0x7f0b2003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0b200387b0 0x7f0b2003ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c006fd0 tx=0x7f0b2c006e40 comp rx=0 tx=0).stop 2023-07-20T02:34:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 msgr2=0x7f0b381956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b381956b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0b28006010 tx=0x7f0b28004ff0 comp rx=0 tx=0).stop 2023-07-20T02:34:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 shutdown_connections 2023-07-20T02:34:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f0b200387b0 0x7f0b2003ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 --2- 172.21.15.17:0/1679742136 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0b38105180 0x7f0b381956b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 >> 172.21.15.17:0/1679742136 conn(0x7f0b38074c70 msgr2=0x7f0b38102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:11.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 shutdown_connections 2023-07-20T02:34:11.816 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:11.823+0000 7f0b3ceb6700 1 -- 172.21.15.17:0/1679742136 wait complete. 2023-07-20T02:34:12.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T02:34:12.691 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:12.888 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:12.889 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: Updating smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:34:12.889 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:12.889 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:12.889 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:12.889 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:12.889 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:12.889 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.875+0000 7f3b5680d700 1 -- 172.21.15.17:0/1001595092 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 msgr2=0x7f3b50105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.875+0000 7f3b5680d700 1 --2- 172.21.15.17:0/1001595092 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b50105560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3b40009910 tx=0x7f3b40009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:13.868 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.875+0000 7f3b5680d700 1 -- 172.21.15.17:0/1001595092 shutdown_connections 2023-07-20T02:34:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.875+0000 7f3b5680d700 1 --2- 172.21.15.17:0/1001595092 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b50105560 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.875+0000 7f3b5680d700 1 -- 172.21.15.17:0/1001595092 >> 172.21.15.17:0/1001595092 conn(0x7f3b50074c70 msgr2=0x7f3b50075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.875+0000 7f3b5680d700 1 -- 172.21.15.17:0/1001595092 shutdown_connections 2023-07-20T02:34:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.875+0000 7f3b5680d700 1 -- 172.21.15.17:0/1001595092 wait complete. 2023-07-20T02:34:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.876+0000 7f3b5680d700 1 Processor -- start 2023-07-20T02:34:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.876+0000 7f3b5680d700 1 -- start start 2023-07-20T02:34:13.871 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.876+0000 7f3b5680d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b501956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:13.871 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.876+0000 7f3b5680d700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b50195bf0 con 0x7f3b50105180 2023-07-20T02:34:13.871 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.876+0000 7f3b4ffff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b501956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:13.871 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b4ffff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b501956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42674/0 (socket says 172.21.15.17:42674) 2023-07-20T02:34:13.871 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b4ffff700 1 -- 172.21.15.17:0/4054374833 learned_addr learned my addr 172.21.15.17:0/4054374833 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:34:13.872 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b4ffff700 1 -- 172.21.15.17:0/4054374833 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400095f0 con 0x7f3b50105180 2023-07-20T02:34:13.872 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b4ffff700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b501956b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b40004f40 tx=0x7f3b40005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:13.872 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b4d7fa700 1 -- 172.21.15.17:0/4054374833 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b4001d070 con 0x7f3b50105180 2023-07-20T02:34:13.872 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b4d7fa700 1 -- 172.21.15.17:0/4054374833 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b40004370 con 0x7f3b50105180 2023-07-20T02:34:13.872 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b4d7fa700 1 -- 172.21.15.17:0/4054374833 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b40003970 con 0x7f3b50105180 2023-07-20T02:34:13.872 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b50195e50 con 0x7f3b50105180 2023-07-20T02:34:13.872 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.877+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b501962d0 con 0x7f3b50105180 2023-07-20T02:34:13.873 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.878+0000 7f3b4d7fa700 1 -- 172.21.15.17:0/4054374833 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3b40003c30 con 0x7f3b50105180 2023-07-20T02:34:13.873 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.878+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b50061da0 con 0x7f3b50105180 2023-07-20T02:34:13.873 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.878+0000 7f3b4d7fa700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b380387b0 0x7f3b3803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:13.873 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.878+0000 7f3b4d7fa700 1 -- 172.21.15.17:0/4054374833 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3b4004c750 con 0x7f3b50105180 2023-07-20T02:34:13.873 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.878+0000 7f3b4f7fe700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b380387b0 0x7f3b3803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:13.873 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.879+0000 7f3b4f7fe700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b380387b0 0x7f3b3803ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3b44006fd0 tx=0x7f3b44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:13.875 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:13.883+0000 7f3b4d7fa700 1 -- 172.21.15.17:0/4054374833 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3b4002aba0 con 0x7f3b50105180 2023-07-20T02:34:13.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:13 smithi017 ceph-mon[127345]: Added host smithi151 2023-07-20T02:34:13.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:14.079 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.087+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b50104450 con 0x7f3b380387b0 2023-07-20T02:34:14.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.088+0000 7f3b4d7fa700 1 -- 172.21.15.17:0/4054374833 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f3b50104450 con 0x7f3b380387b0 2023-07-20T02:34:14.103 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:34:14.104 INFO:teuthology.orchestra.run.smithi017.stdout:[{"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}, {"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}] 2023-07-20T02:34:14.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b380387b0 msgr2=0x7f3b3803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:14.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b380387b0 0x7f3b3803ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3b44006fd0 tx=0x7f3b44006e40 comp rx=0 tx=0).stop 2023-07-20T02:34:14.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 msgr2=0x7f3b501956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:14.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b501956b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b40004f40 tx=0x7f3b40005020 comp rx=0 tx=0).stop 2023-07-20T02:34:14.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 shutdown_connections 2023-07-20T02:34:14.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b380387b0 0x7f3b3803ac70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:14.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 --2- 172.21.15.17:0/4054374833 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b50105180 0x7f3b501956b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:14.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 >> 172.21.15.17:0/4054374833 conn(0x7f3b50074c70 msgr2=0x7f3b50102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:14.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 shutdown_connections 2023-07-20T02:34:14.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:14.091+0000 7f3b5680d700 1 -- 172.21.15.17:0/4054374833 wait complete. 2023-07-20T02:34:14.448 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-20T02:34:14.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph osd crush tunables default 2023-07-20T02:34:14.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:14 smithi017 ceph-mon[127345]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:14.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:14 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:15.011 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:15.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:15 smithi017 ceph-mon[127345]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:34:16.172 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.180+0000 7fa4ea241700 1 -- 172.21.15.17:0/1290060798 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 msgr2=0x7fa4e41017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:16.172 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.180+0000 7fa4ea241700 1 --2- 172.21.15.17:0/1290060798 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41017e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009910 tx=0x7fa4d4009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:16.172 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.180+0000 7fa4ea241700 1 -- 172.21.15.17:0/1290060798 shutdown_connections 2023-07-20T02:34:16.172 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.180+0000 7fa4ea241700 1 --2- 172.21.15.17:0/1290060798 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41017e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:16.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.180+0000 7fa4ea241700 1 -- 172.21.15.17:0/1290060798 >> 172.21.15.17:0/1290060798 conn(0x7fa4e40fabf0 msgr2=0x7fa4e40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:16.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.180+0000 7fa4ea241700 1 -- 172.21.15.17:0/1290060798 shutdown_connections 2023-07-20T02:34:16.173 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.180+0000 7fa4ea241700 1 -- 172.21.15.17:0/1290060798 wait complete. 2023-07-20T02:34:16.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4ea241700 1 Processor -- start 2023-07-20T02:34:16.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4ea241700 1 -- start start 2023-07-20T02:34:16.177 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4ea241700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41956d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:16.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4ea241700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e4195c10 con 0x7fa4e4101400 2023-07-20T02:34:16.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4e37fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:16.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4e37fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41956d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42680/0 (socket says 172.21.15.17:42680) 2023-07-20T02:34:16.178 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4e37fe700 1 -- 172.21.15.17:0/790098867 learned_addr learned my addr 172.21.15.17:0/790098867 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4e37fe700 1 -- 172.21.15.17:0/790098867 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d40095f0 con 0x7fa4e4101400 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.181+0000 7fa4e37fe700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41956d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4005850 tx=0x7fa4d4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.182+0000 7fa4e0ff9700 1 -- 172.21.15.17:0/790098867 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4d401d070 con 0x7fa4e4101400 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.182+0000 7fa4e0ff9700 1 -- 172.21.15.17:0/790098867 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4d4004370 con 0x7fa4e4101400 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.182+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e4195e10 con 0x7fa4e4101400 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.182+0000 7fa4e0ff9700 1 -- 172.21.15.17:0/790098867 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4d4003ab0 con 0x7fa4e4101400 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.182+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e4196230 con 0x7fa4e4101400 2023-07-20T02:34:16.179 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.183+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e4061da0 con 0x7fa4e4101400 2023-07-20T02:34:16.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.183+0000 7fa4e0ff9700 1 -- 172.21.15.17:0/790098867 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa4d400b6f0 con 0x7fa4e4101400 2023-07-20T02:34:16.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.183+0000 7fa4e0ff9700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa4d00387b0 0x7fa4d003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:16.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.183+0000 7fa4e0ff9700 1 -- 172.21.15.17:0/790098867 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa4d404c520 con 0x7fa4e4101400 2023-07-20T02:34:16.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.183+0000 7fa4e2ffd700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa4d00387b0 0x7fa4d003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:16.180 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.184+0000 7fa4e2ffd700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa4d00387b0 0x7fa4d003ac70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc006fd0 tx=0x7fa4cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:16.181 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.188+0000 7fa4e0ff9700 1 -- 172.21.15.17:0/790098867 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa4d402a590 con 0x7fa4e4101400 2023-07-20T02:34:16.383 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.391+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa4e4107350 con 0x7fa4e4101400 2023-07-20T02:34:16.567 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.575+0000 7fa4e0ff9700 1 -- 172.21.15.17:0/790098867 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa4d4026070 con 0x7fa4e4101400 2023-07-20T02:34:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa4d00387b0 msgr2=0x7fa4d003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa4d00387b0 0x7fa4d003ac70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc006fd0 tx=0x7fa4cc006e40 comp rx=0 tx=0).stop 2023-07-20T02:34:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 msgr2=0x7fa4e41956d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41956d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4005850 tx=0x7fa4d4004ff0 comp rx=0 tx=0).stop 2023-07-20T02:34:16.570 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 shutdown_connections 2023-07-20T02:34:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa4d00387b0 0x7fa4d003ac70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 --2- 172.21.15.17:0/790098867 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa4e4101400 0x7fa4e41956d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 >> 172.21.15.17:0/790098867 conn(0x7fa4e40fabf0 msgr2=0x7fa4e4104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 shutdown_connections 2023-07-20T02:34:16.571 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:16.578+0000 7fa4ea241700 1 -- 172.21.15.17:0/790098867 wait complete. 2023-07-20T02:34:16.572 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2023-07-20T02:34:16.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:16 smithi017 ceph-mon[127345]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:16.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:16 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/790098867' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T02:34:16.956 INFO:tasks.cephadm:Adding mon.a on smithi017 2023-07-20T02:34:16.956 INFO:tasks.cephadm:Adding mon.b on smithi146 2023-07-20T02:34:16.956 INFO:tasks.cephadm:Adding mon.c on smithi151 2023-07-20T02:34:16.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch apply mon '3;smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c' 2023-07-20T02:34:17.554 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:17.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/790098867' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T02:34:17.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:17.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:17.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:17.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:17.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:17.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:34:17.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:17.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:17 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:18.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 -- 172.21.15.151:0/4208447655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 msgr2=0x7f437c105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 --2- 172.21.15.151:0/4208447655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c105560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4370009910 tx=0x7f4370009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 -- 172.21.15.151:0/4208447655 shutdown_connections 2023-07-20T02:34:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 --2- 172.21.15.151:0/4208447655 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c105560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 -- 172.21.15.151:0/4208447655 >> 172.21.15.151:0/4208447655 conn(0x7f437c074c70 msgr2=0x7f437c075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 -- 172.21.15.151:0/4208447655 shutdown_connections 2023-07-20T02:34:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 -- 172.21.15.151:0/4208447655 wait complete. 2023-07-20T02:34:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.738+0000 7f4381cbb700 1 Processor -- start 2023-07-20T02:34:18.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.739+0000 7f4381cbb700 1 -- start start 2023-07-20T02:34:18.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.739+0000 7f4381cbb700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:18.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.739+0000 7f4381cbb700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c196a40 con 0x7f437c105180 2023-07-20T02:34:18.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.739+0000 7f437b7fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:18.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.739+0000 7f437b7fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.151:46306/0 (socket says 172.21.15.151:46306) 2023-07-20T02:34:18.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.739+0000 7f437b7fe700 1 -- 172.21.15.151:0/2641282850 learned_addr learned my addr 172.21.15.151:0/2641282850 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:34:18.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.739+0000 7f437b7fe700 1 -- 172.21.15.151:0/2641282850 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700095f0 con 0x7f437c105180 2023-07-20T02:34:18.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.740+0000 7f437b7fe700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c196500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4370004f40 tx=0x7f4370005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:18.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.740+0000 7f4378ff9700 1 -- 172.21.15.151:0/2641282850 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f437001d070 con 0x7f437c105180 2023-07-20T02:34:18.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.740+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c196ca0 con 0x7f437c105180 2023-07-20T02:34:18.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.740+0000 7f4378ff9700 1 -- 172.21.15.151:0/2641282850 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4370004370 con 0x7f437c105180 2023-07-20T02:34:18.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.740+0000 7f4378ff9700 1 -- 172.21.15.151:0/2641282850 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4370003970 con 0x7f437c105180 2023-07-20T02:34:18.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.740+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c197120 con 0x7f437c105180 2023-07-20T02:34:18.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.741+0000 7f4378ff9700 1 -- 172.21.15.151:0/2641282850 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4370003c30 con 0x7f437c105180 2023-07-20T02:34:18.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.741+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c197360 con 0x7f437c105180 2023-07-20T02:34:18.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.741+0000 7f4378ff9700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43640387b0 0x7f436403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:18.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.741+0000 7f4378ff9700 1 -- 172.21.15.151:0/2641282850 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f437004c4a0 con 0x7f437c105180 2023-07-20T02:34:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.741+0000 7f437affd700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43640387b0 0x7f436403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:18.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.742+0000 7f437affd700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43640387b0 0x7f436403ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f436c006fd0 tx=0x7f436c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.746+0000 7f4378ff9700 1 -- 172.21.15.151:0/2641282850 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f437002a590 con 0x7f437c105180 2023-07-20T02:34:18.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.942+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f437c060bb0 con 0x7f43640387b0 2023-07-20T02:34:18.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.949+0000 7f4378ff9700 1 -- 172.21.15.151:0/2641282850 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f437c060bb0 con 0x7f43640387b0 2023-07-20T02:34:18.942 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mon update... 2023-07-20T02:34:18.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43640387b0 msgr2=0x7f436403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:18.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43640387b0 0x7f436403ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f436c006fd0 tx=0x7f436c006e40 comp rx=0 tx=0).stop 2023-07-20T02:34:18.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 msgr2=0x7f437c196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:18.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c196500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4370004f40 tx=0x7f4370005020 comp rx=0 tx=0).stop 2023-07-20T02:34:18.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 shutdown_connections 2023-07-20T02:34:18.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43640387b0 0x7f436403ac70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:18.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 --2- 172.21.15.151:0/2641282850 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f437c105180 0x7f437c196500 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:18.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.951+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 >> 172.21.15.151:0/2641282850 conn(0x7f437c074c70 msgr2=0x7f437c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:18.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.952+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 shutdown_connections 2023-07-20T02:34:18.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:18.952+0000 7f4381cbb700 1 -- 172.21.15.151:0/2641282850 wait complete. 2023-07-20T02:34:19.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:19.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: Updating smithi151:/etc/ceph/ceph.conf 2023-07-20T02:34:19.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:19.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:19.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:34:19.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:19.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:19.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:18 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:19.424 DEBUG:teuthology.orchestra.run.smithi146:mon.b> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.b.service 2023-07-20T02:34:19.427 DEBUG:teuthology.orchestra.run.smithi151:mon.c> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.c.service 2023-07-20T02:34:19.430 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T02:34:19.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T02:34:19.460 INFO:journalctl@ceph.mon.c.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:34:19.462 INFO:journalctl@ceph.mon.b.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:34:20.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:20.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:20.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c;count:3 2023-07-20T02:34:20.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:20.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:20.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:20.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:20.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:20.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:20.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:20.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:19 smithi017 ceph-mon[127345]: Deploying daemon mon.c on smithi151 2023-07-20T02:34:20.352 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:22.103 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:21 smithi151 systemd[1]: Starting Ceph mon.c for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:34:22.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.127+0000 7f4b47a95700 1 -- 172.21.15.151:0/3293679914 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 msgr2=0x7f4b40100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:22.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.127+0000 7f4b47a95700 1 --2- 172.21.15.151:0/3293679914 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b40100830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009910 tx=0x7f4b3c009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:22.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.127+0000 7f4b47a95700 1 -- 172.21.15.151:0/3293679914 shutdown_connections 2023-07-20T02:34:22.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.127+0000 7f4b47a95700 1 --2- 172.21.15.151:0/3293679914 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b40100830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:22.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.127+0000 7f4b47a95700 1 -- 172.21.15.151:0/3293679914 >> 172.21.15.151:0/3293679914 conn(0x7f4b400fbd00 msgr2=0x7f4b400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:22.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.127+0000 7f4b47a95700 1 -- 172.21.15.151:0/3293679914 shutdown_connections 2023-07-20T02:34:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.127+0000 7f4b47a95700 1 -- 172.21.15.151:0/3293679914 wait complete. 2023-07-20T02:34:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b47a95700 1 Processor -- start 2023-07-20T02:34:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b47a95700 1 -- start start 2023-07-20T02:34:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b47a95700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b401956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b47a95700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b40195c00 con 0x7f4b40100450 2023-07-20T02:34:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b45831700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b401956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b45831700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b401956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.151:46312/0 (socket says 172.21.15.151:46312) 2023-07-20T02:34:22.123 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b45831700 1 -- 172.21.15.151:0/769794220 learned_addr learned my addr 172.21.15.151:0/769794220 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:34:22.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.128+0000 7f4b45831700 1 -- 172.21.15.151:0/769794220 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0095f0 con 0x7f4b40100450 2023-07-20T02:34:22.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.129+0000 7f4b45831700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b401956c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c004f40 tx=0x7f4b3c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:22.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.129+0000 7f4b36ffd700 1 -- 172.21.15.151:0/769794220 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b3c01d070 con 0x7f4b40100450 2023-07-20T02:34:22.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.129+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b40195e00 con 0x7f4b40100450 2023-07-20T02:34:22.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.129+0000 7f4b36ffd700 1 -- 172.21.15.151:0/769794220 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b3c004370 con 0x7f4b40100450 2023-07-20T02:34:22.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.129+0000 7f4b36ffd700 1 -- 172.21.15.151:0/769794220 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b3c0039b0 con 0x7f4b40100450 2023-07-20T02:34:22.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.129+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b40196220 con 0x7f4b40100450 2023-07-20T02:34:22.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.130+0000 7f4b36ffd700 1 -- 172.21.15.151:0/769794220 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4b3c003c70 con 0x7f4b40100450 2023-07-20T02:34:22.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.130+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b40061da0 con 0x7f4b40100450 2023-07-20T02:34:22.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.130+0000 7f4b36ffd700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f4b2c0387b0 0x7f4b2c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:22.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.130+0000 7f4b36ffd700 1 -- 172.21.15.151:0/769794220 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4b3c04c790 con 0x7f4b40100450 2023-07-20T02:34:22.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.130+0000 7f4b45030700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f4b2c0387b0 0x7f4b2c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:22.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.131+0000 7f4b45030700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f4b2c0387b0 0x7f4b2c03ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b30006fd0 tx=0x7f4b30006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:22.128 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.136+0000 7f4b36ffd700 1 -- 172.21.15.151:0/769794220 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4b3c029360 con 0x7f4b40100450 2023-07-20T02:34:22.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:21 smithi017 ceph-mon[127345]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.417+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b401081a0 con 0x7f4b40100450 2023-07-20T02:34:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.417+0000 7f4b36ffd700 1 -- 172.21.15.151:0/769794220 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f4b3c026030 con 0x7f4b40100450 2023-07-20T02:34:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:34:22.412 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","modified":"2023-07-20T02:32:48.115052Z","created":"2023-07-20T02:32:48.115052Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-20T02:34:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f4b2c0387b0 msgr2=0x7f4b2c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f4b2c0387b0 0x7f4b2c03ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b30006fd0 tx=0x7f4b30006e40 comp rx=0 tx=0).stop 2023-07-20T02:34:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 msgr2=0x7f4b401956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b401956c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c004f40 tx=0x7f4b3c005020 comp rx=0 tx=0).stop 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 shutdown_connections 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f4b2c0387b0 0x7f4b2c03ac70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 --2- 172.21.15.151:0/769794220 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4b40100450 0x7f4b401956c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 >> 172.21.15.151:0/769794220 conn(0x7f4b400fbd00 msgr2=0x7f4b401046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 shutdown_connections 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:22.420+0000 7f4b47a95700 1 -- 172.21.15.151:0/769794220 wait complete. 2023-07-20T02:34:22.414 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 1 2023-07-20T02:34:22.559 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 podman[129110]: 2023-07-20T02:34:22.815 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T02:34:22.816 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T02:34:22.816 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: pidfile_write: ignore empty --pid-file 2023-07-20T02:34:22.816 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: load: jerasure load: lrc 2023-07-20T02:34:22.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: RocksDB version: 7.9.2 2023-07-20T02:34:22.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Git sha 0 2023-07-20T02:34:22.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T02:34:22.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: DB SUMMARY 2023-07-20T02:34:22.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: DB Session ID: XEOM3C8ZMBJJ7U1ER1JN 2023-07-20T02:34:22.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: CURRENT file: CURRENT 2023-07-20T02:34:22.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.error_if_exists: 0 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.create_if_missing: 0 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.paranoid_checks: 1 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T02:34:22.818 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T02:34:22.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T02:34:22.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.env: 0x55e81241bc60 2023-07-20T02:34:22.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T02:34:22.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.info_log: 0x55e812d552c0 2023-07-20T02:34:22.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T02:34:22.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.statistics: (nil) 2023-07-20T02:34:22.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.use_fsync: 0 2023-07-20T02:34:22.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_log_file_size: 0 2023-07-20T02:34:22.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T02:34:22.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T02:34:22.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T02:34:22.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T02:34:22.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.allow_fallocate: 1 2023-07-20T02:34:22.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.use_direct_reads: 0 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.db_log_dir: 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.wal_dir: 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T02:34:22.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.write_buffer_manager: 0x55e812c763c0 2023-07-20T02:34:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T02:34:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T02:34:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T02:34:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T02:34:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T02:34:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T02:34:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T02:34:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T02:34:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.unordered_write: 0 2023-07-20T02:34:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T02:34:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T02:34:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T02:34:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T02:34:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.row_cache: None 2023-07-20T02:34:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.wal_filter: None 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.two_write_queues: 0 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.wal_compression: 0 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.atomic_flush: 0 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T02:34:22.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.log_readahead_size: 0 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T02:34:22.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T02:34:22.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T02:34:22.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_background_jobs: 2 2023-07-20T02:34:22.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_background_compactions: -1 2023-07-20T02:34:22.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_subcompactions: 1 2023-07-20T02:34:22.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T02:34:22.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T02:34:22.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T02:34:22.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T02:34:22.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T02:34:22.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T02:34:22.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T02:34:22.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T02:34:22.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_open_files: -1 2023-07-20T02:34:22.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T02:34:22.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T02:34:22.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T02:34:22.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T02:34:22.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_background_flushes: -1 2023-07-20T02:34:22.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Compression algorithms supported: 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kZSTD supported: 0 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kXpressCompression supported: 0 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kZlibCompression supported: 1 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kSnappyCompression supported: 1 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kLZ4Compression supported: 1 2023-07-20T02:34:22.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: kBZip2Compression supported: 0 2023-07-20T02:34:22.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T02:34:22.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T02:34:22.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-20T02:34:22.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T02:34:22.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T02:34:22.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.merge_operator: 2023-07-20T02:34:22.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_filter: None 2023-07-20T02:34:22.833 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T02:34:22.833 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T02:34:22.833 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T02:34:22.833 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T02:34:22.833 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e812d553c0) 2023-07-20T02:34:22.833 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks: 1 2023-07-20T02:34:22.833 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_top_level_index_and_filter: 1 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_type: 0 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_index_type: 0 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_shortening: 1 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: checksum: 4 2023-07-20T02:34:22.834 INFO:journalctl@ceph.mon.c.smithi151.stdout: no_block_cache: 0 2023-07-20T02:34:22.835 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache: 0x55e812d15350 2023-07-20T02:34:22.835 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_name: BinnedLRUCache 2023-07-20T02:34:22.835 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_options: 2023-07-20T02:34:22.835 INFO:journalctl@ceph.mon.c.smithi151.stdout: capacity : 536870912 2023-07-20T02:34:22.835 INFO:journalctl@ceph.mon.c.smithi151.stdout: num_shard_bits : 4 2023-07-20T02:34:22.835 INFO:journalctl@ceph.mon.c.smithi151.stdout: strict_capacity_limit : 0 2023-07-20T02:34:22.835 INFO:journalctl@ceph.mon.c.smithi151.stdout: high_pri_pool_ratio: 0.000 2023-07-20T02:34:22.836 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_compressed: (nil) 2023-07-20T02:34:22.836 INFO:journalctl@ceph.mon.c.smithi151.stdout: persistent_cache: (nil) 2023-07-20T02:34:22.836 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size: 4096 2023-07-20T02:34:22.836 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size_deviation: 10 2023-07-20T02:34:22.836 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_restart_interval: 16 2023-07-20T02:34:22.836 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_block_restart_interval: 1 2023-07-20T02:34:22.836 INFO:journalctl@ceph.mon.c.smithi151.stdout: metadata_block_size: 4096 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: partition_filters: 0 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: use_delta_encoding: 1 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: filter_policy: bloomfilter 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: whole_key_filtering: 1 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: verify_compression: 0 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: read_amp_bytes_per_bit: 0 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: format_version: 5 2023-07-20T02:34:22.837 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_index_compression: 1 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_align: 0 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout: max_auto_readahead_size: 262144 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout: prepopulate_block_cache: 0 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout: initial_auto_readahead_size: 8192 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T02:34:22.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression: NoCompression 2023-07-20T02:34:22.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T02:34:22.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T02:34:22.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T02:34:22.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.num_levels: 7 2023-07-20T02:34:22.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T02:34:22.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T02:34:22.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T02:34:22.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T02:34:22.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T02:34:22.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T02:34:22.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T02:34:22.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:34:22.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T02:34:22.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:34:22.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T02:34:22.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T02:34:22.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T02:34:22.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T02:34:22.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T02:34:22.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T02:34:22.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T02:34:22.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T02:34:22.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T02:34:22.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T02:34:22.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T02:34:22.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T02:34:22.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T02:34:22.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T02:34:22.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T02:34:22.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T02:34:22.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.table_properties_collectors: 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.inplace_update_support: 0 2023-07-20T02:34:22.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.bloom_locality: 0 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.max_successive_merges: 0 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T02:34:22.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.ttl: 2592000 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.enable_blob_files: false 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.min_blob_size: 0 2023-07-20T02:34:22.848 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T02:34:22.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T02:34:22.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T02:34:22.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5821fe91-b63d-4d3e-ae7f-80042a4e9d49 2023-07-20T02:34:22.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820462574342, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T02:34:22.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T02:34:22.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820462574752, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689820462, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5821fe91-b63d-4d3e-ae7f-80042a4e9d49", "db_session_id": "XEOM3C8ZMBJJ7U1ER1JN", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T02:34:22.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820462574820, "job": 1, "event": "recovery_finished"} 2023-07-20T02:34:22.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T02:34:22.851 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T02:34:22.851 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e812dae000 2023-07-20T02:34:22.851 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: DB pointer 0x55e812d9a000 2023-07-20T02:34:22.851 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-20T02:34:22.851 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T02:34:22.851 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T02:34:22.851 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** DB Stats ** 2023-07-20T02:34:22.852 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:34:22.852 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T02:34:22.852 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:34:22.852 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:34:22.852 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T02:34:22.852 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:34:22.852 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:34:22.853 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2023-07-20T02:34:22.854 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:34:22.855 INFO:journalctl@ceph.mon.c.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T02:34:22.856 INFO:journalctl@ceph.mon.c.smithi151.stdout: Block cache BinnedLRUCache@0x55e812d15350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-20T02:34:22.856 INFO:journalctl@ceph.mon.c.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T02:34:22.856 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2023-07-20T02:34:22.856 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T02:34:22.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: using public_addr v2:172.21.15.151:0/0 -> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-07-20T02:34:22.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: starting mon.c rank -1 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:34:22.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(???) e0 preinit fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:34:22.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).mds e1 new map 2023-07-20T02:34:22.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-20T02:34:22.857 INFO:journalctl@ceph.mon.c.smithi151.stdout: e1 2023-07-20T02:34:22.857 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T02:34:22.857 INFO:journalctl@ceph.mon.c.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T02:34:22.857 INFO:journalctl@ceph.mon.c.smithi151.stdout: legacy client fscid: -1 2023-07-20T02:34:22.857 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout: No filesystems configured 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T02:34:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Updating smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:22.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Added host smithi151 2023-07-20T02:34:22.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/790098867' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/790098867' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Updating smithi151:/etc/ceph/ceph.conf 2023-07-20T02:34:22.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:22.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c;count:3 2023-07-20T02:34:22.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:22.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:22.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:22.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:22.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:22.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:22.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: Deploying daemon mon.c on smithi151 2023-07-20T02:34:22.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:22.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T02:34:22.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-20T02:34:22.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expand map: {default=false} 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta from 'false' to 'false' 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expanded map: {default=false} 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expand map: {default=info} 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta from 'info' to 'info' 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expanded map: {default=info} 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expand map: {default=daemon} 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T02:34:22.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T02:34:22.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expand map: {default=debug} 2023-07-20T02:34:22.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T02:34:22.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 ceph-mon[129163]: expand_channel_meta expanded map: {default=debug} 2023-07-20T02:34:22.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 bash[129110]: 5fba22a13359b174f7be41b4517da56938d9f8f9dc3f7f44fcd3ddfe2aeea77a 2023-07-20T02:34:22.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:22 smithi151 systemd[1]: Started Ceph mon.c for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:34:23.869 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T02:34:23.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T02:34:24.436 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:34:28.103 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:34:28.103 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: mon.a calling monitor election 2023-07-20T02:34:28.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:28.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: mon.c calling monitor election 2023-07-20T02:34:28.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:28.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:34:28.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:28.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: fsmap 2023-07-20T02:34:28.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: mgrmap e13: x(active, since 42s) 2023-07-20T02:34:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: overall HEALTH_OK 2023-07-20T02:34:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:28.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:34:28.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: mon.a calling monitor election 2023-07-20T02:34:28.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:28.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: mon.c calling monitor election 2023-07-20T02:34:28.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:28.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:28.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:34:28.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:28.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: fsmap 2023-07-20T02:34:28.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:28.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: mgrmap e13: x(active, since 42s) 2023-07-20T02:34:28.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: overall HEALTH_OK 2023-07-20T02:34:28.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:28.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:28.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.726+0000 7f7510aa7700 1 -- 172.21.15.151:0/2483943613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 msgr2=0x7f74f4007820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.726+0000 7f7510aa7700 1 --2- 172.21.15.151:0/2483943613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 0x7f74f4007820 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7500005fa0 tx=0x7f750000ffc0 comp rx=0 tx=0).stop 2023-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.726+0000 7f7510aa7700 1 -- 172.21.15.151:0/2483943613 shutdown_connections 2023-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.726+0000 7f7510aa7700 1 --2- 172.21.15.151:0/2483943613 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 0x7f74f4007820 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.726+0000 7f7510aa7700 1 -- 172.21.15.151:0/2483943613 >> 172.21.15.151:0/2483943613 conn(0x7f750c0fad70 msgr2=0x7f750c0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.726+0000 7f7510aa7700 1 -- 172.21.15.151:0/2483943613 shutdown_connections 2023-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.726+0000 7f7510aa7700 1 -- 172.21.15.151:0/2483943613 wait complete. 2023-07-20T02:34:28.721 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7510aa7700 1 Processor -- start 2023-07-20T02:34:28.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7510aa7700 1 -- start start 2023-07-20T02:34:28.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7510aa7700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 0x7f750c1936c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:28.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7510aa7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 0x7f750c196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:28.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7510aa7700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c194190 con 0x7f750c0ff4a0 2023-07-20T02:34:28.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7510aa7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c1942d0 con 0x7f750c193c00 2023-07-20T02:34:28.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7509d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 0x7f750c196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:28.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7509d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 0x7f750c196fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40684/0 (socket says 172.21.15.151:40684) 2023-07-20T02:34:28.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7509d9b700 1 -- 172.21.15.151:0/937774789 learned_addr learned my addr 172.21.15.151:0/937774789 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:34:28.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7509d9b700 1 -- 172.21.15.151:0/937774789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 msgr2=0x7f750c196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T02:34:28.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7509d9b700 1 -- 172.21.15.151:0/937774789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 msgr2=0x7f750c196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T02:34:28.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f750a59c700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 0x7f750c1936c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:28.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7509d9b700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 0x7f750c196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T02:34:28.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.728+0000 7f7509d9b700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 0x7f750c196fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:34:28.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f750a59c700 1 -- 172.21.15.151:0/937774789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 msgr2=0x7f750c196fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:34:28.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f750a59c700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 0x7f750c196fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:28.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f750a59c700 1 -- 172.21.15.151:0/937774789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750000f9c0 con 0x7f750c0ff4a0 2023-07-20T02:34:28.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f750a59c700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 0x7f750c1936c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7500000c00 tx=0x7f7500004cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:28.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f74fb7fe700 1 -- 172.21.15.151:0/937774789 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f75000053b0 con 0x7f750c0ff4a0 2023-07-20T02:34:28.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f74fb7fe700 1 -- 172.21.15.151:0/937774789 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7500021040 con 0x7f750c0ff4a0 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c197500 con 0x7f750c0ff4a0 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f74fb7fe700 1 -- 172.21.15.151:0/937774789 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7500011470 con 0x7f750c0ff4a0 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.729+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c197a20 con 0x7f750c0ff4a0 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.730+0000 7f74fb7fe700 1 -- 172.21.15.151:0/937774789 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f75000116e0 con 0x7f750c0ff4a0 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.730+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c061da0 con 0x7f750c0ff4a0 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.730+0000 7f74fb7fe700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f74ec0385f0 0x7f74ec03aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.731+0000 7f74fb7fe700 1 -- 172.21.15.151:0/937774789 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f750004dc90 con 0x7f750c0ff4a0 2023-07-20T02:34:28.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.731+0000 7f7509d9b700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f74ec0385f0 0x7f74ec03aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:28.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.731+0000 7f7509d9b700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f74ec0385f0 0x7f74ec03aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009180 tx=0x7f74fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:28.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.737+0000 7f74fb7fe700 1 -- 172.21.15.151:0/937774789 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7500019330 con 0x7f750c0ff4a0 2023-07-20T02:34:28.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.997+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f750c106150 con 0x7f750c0ff4a0 2023-07-20T02:34:28.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:28.998+0000 7f74fb7fe700 1 -- 172.21.15.151:0/937774789 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f7500010bc0 con 0x7f750c0ff4a0 2023-07-20T02:34:28.991 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:34:28.992 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":2,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","modified":"2023-07-20T02:34:22.602809Z","created":"2023-07-20T02:32:48.115052Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T02:34:28.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f74ec0385f0 msgr2=0x7f74ec03aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:28.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f74ec0385f0 0x7f74ec03aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009180 tx=0x7f74fc008040 comp rx=0 tx=0).stop 2023-07-20T02:34:28.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 msgr2=0x7f750c1936c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:28.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 0x7f750c1936c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7500000c00 tx=0x7f7500004cb0 comp rx=0 tx=0).stop 2023-07-20T02:34:28.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 shutdown_connections 2023-07-20T02:34:28.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f74ec0385f0 0x7f74ec03aab0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:28.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f750c0ff4a0 0x7f750c1936c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:28.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 --2- 172.21.15.151:0/937774789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f750c193c00 0x7f750c196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:28.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 >> 172.21.15.151:0/937774789 conn(0x7f750c0fad70 msgr2=0x7f750c0fc260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:28.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 shutdown_connections 2023-07-20T02:34:28.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:29.000+0000 7f7510aa7700 1 -- 172.21.15.151:0/937774789 wait complete. 2023-07-20T02:34:28.995 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 2 2023-07-20T02:34:29.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:28 smithi151 ceph-mon[129163]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:29.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:29.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:29.014 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:29.014 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:28 smithi151 ceph-mon[129163]: Deploying daemon mon.b on smithi146 2023-07-20T02:34:29.014 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:29.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:28 smithi017 ceph-mon[127345]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:29.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:29.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:29.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:29.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:28 smithi017 ceph-mon[127345]: Deploying daemon mon.b on smithi146 2023-07-20T02:34:29.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:30.103 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:29 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/937774789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:34:30.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:29 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/937774789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:34:30.397 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T02:34:30.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T02:34:30.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:30 smithi146 systemd[1]: Starting Ceph mon.b for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:34:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:34:31.037 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:30 smithi146 podman[128801]: 2023-07-20T02:34:31.038 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T02:34:31.038 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T02:34:31.038 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 bash[128801]: f18ca6c00f64a3ed5247fa0f5acdaff0164cf2a6152da7e729bde2968d76594b 2023-07-20T02:34:31.038 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pidfile_write: ignore empty --pid-file 2023-07-20T02:34:31.038 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: load: jerasure load: lrc 2023-07-20T02:34:31.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: RocksDB version: 7.9.2 2023-07-20T02:34:31.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Git sha 0 2023-07-20T02:34:31.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T02:34:31.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: DB SUMMARY 2023-07-20T02:34:31.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: DB Session ID: Y65XC3N6NCPX7GJPCHOV 2023-07-20T02:34:31.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: CURRENT file: CURRENT 2023-07-20T02:34:31.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T02:34:31.040 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T02:34:31.040 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-20T02:34:31.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-20T02:34:31.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.error_if_exists: 0 2023-07-20T02:34:31.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.create_if_missing: 0 2023-07-20T02:34:31.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.paranoid_checks: 1 2023-07-20T02:34:31.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T02:34:31.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T02:34:31.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T02:34:31.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.env: 0x5606368e7c60 2023-07-20T02:34:31.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T02:34:31.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.info_log: 0x560637de52c0 2023-07-20T02:34:31.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T02:34:31.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.statistics: (nil) 2023-07-20T02:34:31.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.use_fsync: 0 2023-07-20T02:34:31.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_log_file_size: 0 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.allow_fallocate: 1 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T02:34:31.043 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.use_direct_reads: 0 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.db_log_dir: 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.wal_dir: 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T02:34:31.044 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T02:34:31.045 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T02:34:31.045 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T02:34:31.045 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T02:34:31.045 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T02:34:31.045 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.write_buffer_manager: 0x560637d063c0 2023-07-20T02:34:31.046 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T02:34:31.046 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T02:34:31.046 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T02:34:31.046 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T02:34:31.046 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T02:34:31.046 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T02:34:31.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T02:34:31.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T02:34:31.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.unordered_write: 0 2023-07-20T02:34:31.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T02:34:31.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T02:34:31.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T02:34:31.048 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T02:34:31.048 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.row_cache: None 2023-07-20T02:34:31.048 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.wal_filter: None 2023-07-20T02:34:31.048 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T02:34:31.048 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.two_write_queues: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.wal_compression: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.atomic_flush: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T02:34:31.049 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.log_readahead_size: 0 2023-07-20T02:34:31.050 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T02:34:31.050 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T02:34:31.050 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T02:34:31.050 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T02:34:31.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T02:34:31.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T02:34:31.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T02:34:31.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_background_jobs: 2 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_background_compactions: -1 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_subcompactions: 1 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T02:34:31.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T02:34:31.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T02:34:31.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T02:34:31.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_open_files: -1 2023-07-20T02:34:31.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T02:34:31.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T02:34:31.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T02:34:31.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_background_flushes: -1 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Compression algorithms supported: 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kZSTD supported: 0 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kXpressCompression supported: 0 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kZlibCompression supported: 1 2023-07-20T02:34:31.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kSnappyCompression supported: 1 2023-07-20T02:34:31.056 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kLZ4Compression supported: 1 2023-07-20T02:34:31.056 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: kBZip2Compression supported: 0 2023-07-20T02:34:31.056 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T02:34:31.056 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T02:34:31.056 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-20T02:34:31.057 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T02:34:31.057 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T02:34:31.057 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.merge_operator: 2023-07-20T02:34:31.057 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_filter: None 2023-07-20T02:34:31.057 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T02:34:31.057 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T02:34:31.057 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T02:34:31.058 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T02:34:31.058 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560637de53c0) 2023-07-20T02:34:31.058 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks: 1 2023-07-20T02:34:31.058 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T02:34:31.058 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T02:34:31.058 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_top_level_index_and_filter: 1 2023-07-20T02:34:31.058 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_type: 0 2023-07-20T02:34:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_index_type: 0 2023-07-20T02:34:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_shortening: 1 2023-07-20T02:34:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T02:34:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout: checksum: 4 2023-07-20T02:34:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout: no_block_cache: 0 2023-07-20T02:34:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache: 0x560637da5350 2023-07-20T02:34:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_name: BinnedLRUCache 2023-07-20T02:34:31.060 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_options: 2023-07-20T02:34:31.060 INFO:journalctl@ceph.mon.b.smithi146.stdout: capacity : 536870912 2023-07-20T02:34:31.060 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_shard_bits : 4 2023-07-20T02:34:31.060 INFO:journalctl@ceph.mon.b.smithi146.stdout: strict_capacity_limit : 0 2023-07-20T02:34:31.060 INFO:journalctl@ceph.mon.b.smithi146.stdout: high_pri_pool_ratio: 0.000 2023-07-20T02:34:31.060 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_compressed: (nil) 2023-07-20T02:34:31.061 INFO:journalctl@ceph.mon.b.smithi146.stdout: persistent_cache: (nil) 2023-07-20T02:34:31.061 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size: 4096 2023-07-20T02:34:31.061 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size_deviation: 10 2023-07-20T02:34:31.061 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_restart_interval: 16 2023-07-20T02:34:31.061 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_block_restart_interval: 1 2023-07-20T02:34:31.061 INFO:journalctl@ceph.mon.b.smithi146.stdout: metadata_block_size: 4096 2023-07-20T02:34:31.061 INFO:journalctl@ceph.mon.b.smithi146.stdout: partition_filters: 0 2023-07-20T02:34:31.062 INFO:journalctl@ceph.mon.b.smithi146.stdout: use_delta_encoding: 1 2023-07-20T02:34:31.062 INFO:journalctl@ceph.mon.b.smithi146.stdout: filter_policy: bloomfilter 2023-07-20T02:34:31.062 INFO:journalctl@ceph.mon.b.smithi146.stdout: whole_key_filtering: 1 2023-07-20T02:34:31.062 INFO:journalctl@ceph.mon.b.smithi146.stdout: verify_compression: 0 2023-07-20T02:34:31.062 INFO:journalctl@ceph.mon.b.smithi146.stdout: read_amp_bytes_per_bit: 0 2023-07-20T02:34:31.062 INFO:journalctl@ceph.mon.b.smithi146.stdout: format_version: 5 2023-07-20T02:34:31.062 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_index_compression: 1 2023-07-20T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_align: 0 2023-07-20T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi146.stdout: max_auto_readahead_size: 262144 2023-07-20T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi146.stdout: prepopulate_block_cache: 0 2023-07-20T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi146.stdout: initial_auto_readahead_size: 8192 2023-07-20T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T02:34:31.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T02:34:31.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression: NoCompression 2023-07-20T02:34:31.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T02:34:31.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T02:34:31.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T02:34:31.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.num_levels: 7 2023-07-20T02:34:31.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T02:34:31.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T02:34:31.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T02:34:31.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T02:34:31.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T02:34:31.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T02:34:31.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T02:34:31.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:34:31.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T02:34:31.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T02:34:31.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:34:31.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:34:31.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T02:34:31.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T02:34:31.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T02:34:31.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T02:34:31.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:34:31.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:34:31.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T02:34:31.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T02:34:31.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:34:31.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T02:34:31.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T02:34:31.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T02:34:31.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T02:34:31.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T02:34:31.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T02:34:31.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T02:34:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T02:34:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T02:34:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T02:34:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T02:34:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T02:34:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T02:34:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T02:34:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T02:34:31.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T02:34:31.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T02:34:31.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T02:34:31.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T02:34:31.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T02:34:31.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T02:34:31.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T02:34:31.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T02:34:31.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T02:34:31.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T02:34:31.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.table_properties_collectors: 2023-07-20T02:34:31.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.inplace_update_support: 0 2023-07-20T02:34:31.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T02:34:31.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T02:34:31.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T02:34:31.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T02:34:31.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.bloom_locality: 0 2023-07-20T02:34:31.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.max_successive_merges: 0 2023-07-20T02:34:31.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T02:34:31.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T02:34:31.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T02:34:31.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T02:34:31.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.ttl: 2592000 2023-07-20T02:34:31.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T02:34:31.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T02:34:31.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T02:34:31.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.enable_blob_files: false 2023-07-20T02:34:31.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.min_blob_size: 0 2023-07-20T02:34:31.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T02:34:31.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T02:34:31.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T02:34:31.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T02:34:31.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T02:34:31.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T02:34:31.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T02:34:31.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T02:34:31.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T02:34:31.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T02:34:31.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ffa436ae-2df1-41f3-90ba-870f25611e6e 2023-07-20T02:34:31.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820471021256, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T02:34:31.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T02:34:31.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820471021823, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689820471, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ffa436ae-2df1-41f3-90ba-870f25611e6e", "db_session_id": "Y65XC3N6NCPX7GJPCHOV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T02:34:31.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689820471021914, "job": 1, "event": "recovery_finished"} 2023-07-20T02:34:31.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T02:34:31.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T02:34:31.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560637e3e000 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: DB pointer 0x560637e2a000 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** DB Stats ** 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:34:31.078 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:34:31.079 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T02:34:31.079 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:34:31.079 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:34:31.079 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2023-07-20T02:34:31.079 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2023-07-20T02:34:31.079 INFO:journalctl@ceph.mon.b.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:34:31.079 INFO:journalctl@ceph.mon.b.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T02:34:31.080 INFO:journalctl@ceph.mon.b.smithi146.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:34:31.080 INFO:journalctl@ceph.mon.b.smithi146.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:34:31.080 INFO:journalctl@ceph.mon.b.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:34:31.080 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2023-07-20T02:34:31.080 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2023-07-20T02:34:31.080 INFO:journalctl@ceph.mon.b.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:34:31.081 INFO:journalctl@ceph.mon.b.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T02:34:31.081 INFO:journalctl@ceph.mon.b.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:34:31.081 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2023-07-20T02:34:31.081 INFO:journalctl@ceph.mon.b.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T02:34:31.081 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2023-07-20T02:34:31.081 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:34:31.081 INFO:journalctl@ceph.mon.b.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T02:34:31.082 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache BinnedLRUCache@0x560637da5350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T02:34:31.083 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T02:34:31.083 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2023-07-20T02:34:31.083 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T02:34:31.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-20T02:34:31.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: using public_addr v2:172.21.15.146:0/0 -> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2023-07-20T02:34:31.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: starting mon.b rank -1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:34:31.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(???) e0 preinit fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:34:31.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).mds e1 new map 2023-07-20T02:34:31.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-20T02:34:31.084 INFO:journalctl@ceph.mon.b.smithi146.stdout: e1 2023-07-20T02:34:31.084 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T02:34:31.084 INFO:journalctl@ceph.mon.b.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T02:34:31.084 INFO:journalctl@ceph.mon.b.smithi146.stdout: legacy client fscid: -1 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout: No filesystems configured 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T02:34:31.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T02:34:31.103 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:30 smithi151 ceph-mon[129163]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:30 smithi017 ceph-mon[127345]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.310 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T02:34:31.310 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:34:31.310 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Updating smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:31.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Added host smithi151 2023-07-20T02:34:31.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/790098867' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/790098867' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:34:31.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:31.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:31.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Updating smithi151:/etc/ceph/ceph.conf 2023-07-20T02:34:31.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:31.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi146:172.21.15.146=b;smithi151:172.21.15.151=c;count:3 2023-07-20T02:34:31.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:31.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:31.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:31.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:31.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:31.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Deploying daemon mon.c on smithi151 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.a calling monitor election 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:31.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.c calling monitor election 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: fsmap 2023-07-20T02:34:31.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:31.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mgrmap e13: x(active, since 42s) 2023-07-20T02:34:31.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: overall HEALTH_OK 2023-07-20T02:34:31.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:31.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: Deploying daemon mon.b on smithi146 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/937774789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-20T02:34:31.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expand map: {default=false} 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta from 'false' to 'false' 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expanded map: {default=false} 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expand map: {default=info} 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta from 'info' to 'info' 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expanded map: {default=info} 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expand map: {default=daemon} 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T02:34:31.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T02:34:31.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expand map: {default=debug} 2023-07-20T02:34:31.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T02:34:31.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 ceph-mon[128841]: expand_channel_meta expanded map: {default=debug} 2023-07-20T02:34:31.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:31 smithi146 systemd[1]: Started Ceph mon.b for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:34:36.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:36.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: mon.c calling monitor election 2023-07-20T02:34:36.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: mon.a calling monitor election 2023-07-20T02:34:36.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:34:36.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:36.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:36.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:36.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:34:36.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:36.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: fsmap 2023-07-20T02:34:36.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:36.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: mgrmap e13: x(active, since 51s) 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.530+0000 7f7d8c494700 1 -- 172.21.15.151:0/409293043 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 msgr2=0x7f7d70005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:36.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.530+0000 7f7d8c494700 1 --2- 172.21.15.151:0/409293043 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 0x7f7d70005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7d84076160 tx=0x7f7d7400fe60 comp rx=0 tx=0).stop 2023-07-20T02:34:36.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.530+0000 7f7d8c494700 1 -- 172.21.15.151:0/409293043 shutdown_connections 2023-07-20T02:34:36.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.530+0000 7f7d8c494700 1 --2- 172.21.15.151:0/409293043 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 0x7f7d70005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:36.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.530+0000 7f7d8c494700 1 -- 172.21.15.151:0/409293043 >> 172.21.15.151:0/409293043 conn(0x7f7d84074d20 msgr2=0x7f7d84075130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:36.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.530+0000 7f7d8c494700 1 -- 172.21.15.151:0/409293043 shutdown_connections 2023-07-20T02:34:36.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.530+0000 7f7d8c494700 1 -- 172.21.15.151:0/409293043 wait complete. 2023-07-20T02:34:36.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.531+0000 7f7d8c494700 1 Processor -- start 2023-07-20T02:34:36.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.531+0000 7f7d8c494700 1 -- start start 2023-07-20T02:34:36.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8c494700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 0x7f7d8419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:36.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8c494700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 0x7f7d841946c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:36.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8c494700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 0x7f7d84195080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8c494700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8419b1f0 con 0x7f7d700035c0 2023-07-20T02:34:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8c494700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d841a1460 con 0x7f7d84194c00 2023-07-20T02:34:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8c494700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d841a15d0 con 0x7f7d8419ab80 2023-07-20T02:34:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d89a2f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 0x7f7d841946c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d89a2f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 0x7f7d841946c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40710/0 (socket says 172.21.15.151:40710) 2023-07-20T02:34:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d89a2f700 1 -- 172.21.15.151:0/3658866804 learned_addr learned my addr 172.21.15.151:0/3658866804 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:34:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8aa31700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 0x7f7d84195080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d89a2f700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 msgr2=0x7f7d841946c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T02:34:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d89a2f700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 msgr2=0x7f7d841946c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T02:34:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d89a2f700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 0x7f7d841946c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T02:34:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8a230700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 0x7f7d8419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d89a2f700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 0x7f7d841946c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:34:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8aa31700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 msgr2=0x7f7d84195080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-20T02:34:36.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8aa31700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 msgr2=0x7f7d84195080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8aa31700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 0x7f7d84195080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8aa31700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 0x7f7d84195080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8a230700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 msgr2=0x7f7d841946c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8a230700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 0x7f7d841946c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8a230700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 msgr2=0x7f7d84195080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8a230700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 0x7f7d84195080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.532+0000 7f7d8a230700 1 -- 172.21.15.151:0/3658866804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7400f8b0 con 0x7f7d700035c0 2023-07-20T02:34:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.533+0000 7f7d8a230700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 0x7f7d8419a640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d74020040 tx=0x7f7d74012d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:36.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.533+0000 7f7d7b7fe700 1 -- 172.21.15.151:0/3658866804 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d74004130 con 0x7f7d700035c0 2023-07-20T02:34:36.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.533+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d841a1850 con 0x7f7d700035c0 2023-07-20T02:34:36.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.533+0000 7f7d7b7fe700 1 -- 172.21.15.151:0/3658866804 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d74004290 con 0x7f7d700035c0 2023-07-20T02:34:36.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.533+0000 7f7d7b7fe700 1 -- 172.21.15.151:0/3658866804 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d74027850 con 0x7f7d700035c0 2023-07-20T02:34:36.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.533+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d841a1d50 con 0x7f7d700035c0 2023-07-20T02:34:36.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.534+0000 7f7d7b7fe700 1 -- 172.21.15.151:0/3658866804 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f7d74024050 con 0x7f7d700035c0 2023-07-20T02:34:36.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.534+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d84061da0 con 0x7f7d700035c0 2023-07-20T02:34:36.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.534+0000 7f7d7b7fe700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7d68038c30 0x7f7d6803b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:36.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.534+0000 7f7d7b7fe700 1 -- 172.21.15.151:0/3658866804 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7d74000e40 con 0x7f7d700035c0 2023-07-20T02:34:36.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.535+0000 7f7d89a2f700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7d68038c30 0x7f7d6803b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:36.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.535+0000 7f7d89a2f700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7d68038c30 0x7f7d6803b0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009180 tx=0x7f7d80008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:36.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.539+0000 7f7d7b7fe700 1 -- 172.21.15.151:0/3658866804 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7d74010a20 con 0x7f7d700035c0 2023-07-20T02:34:36.604 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:36.604 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: mon.c calling monitor election 2023-07-20T02:34:36.604 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: mon.a calling monitor election 2023-07-20T02:34:36.604 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:34:36.604 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:36.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:36.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:36.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:34:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: fsmap 2023-07-20T02:34:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: mgrmap e13: x(active, since 51s) 2023-07-20T02:34:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T02:34:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T02:34:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T02:34:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2023-07-20T02:34:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:36.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.813+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d84195bd0 con 0x7f7d700035c0 2023-07-20T02:34:36.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.814+0000 7f7d7b7fe700 1 -- 172.21.15.151:0/3658866804 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f7d74022070 con 0x7f7d700035c0 2023-07-20T02:34:36.807 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-07-20T02:34:36.807 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":3,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","modified":"2023-07-20T02:34:31.057184Z","created":"2023-07-20T02:32:48.115052Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T02:34:36.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7d68038c30 msgr2=0x7f7d6803b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:36.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7d68038c30 0x7f7d6803b0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009180 tx=0x7f7d80008040 comp rx=0 tx=0).stop 2023-07-20T02:34:36.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 msgr2=0x7f7d8419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:36.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 0x7f7d8419a640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d74020040 tx=0x7f7d74012d70 comp rx=0 tx=0).stop 2023-07-20T02:34:36.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 shutdown_connections 2023-07-20T02:34:36.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f7d68038c30 0x7f7d6803b0f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:36.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d84194c00 0x7f7d84195080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:36.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f7d700035c0 0x7f7d8419a640 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:36.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 --2- 172.21.15.151:0/3658866804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d8419ab80 0x7f7d841946c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:36.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.816+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 >> 172.21.15.151:0/3658866804 conn(0x7f7d84074d20 msgr2=0x7f7d840fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:36.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.817+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 shutdown_connections 2023-07-20T02:34:36.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:36.817+0000 7f7d8c494700 1 -- 172.21.15.151:0/3658866804 wait complete. 2023-07-20T02:34:36.811 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 3 2023-07-20T02:34:37.231 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-20T02:34:37.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph config generate-minimal-conf 2023-07-20T02:34:37.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:37 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:37.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:37 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/3658866804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:34:37.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:37 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:37.603 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:37 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:37.604 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:37 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/3658866804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:34:37.604 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:37 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:37.809 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:38.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:38.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.c calling monitor election 2023-07-20T02:34:38.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.a calling monitor election 2023-07-20T02:34:38.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:34:38.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:38.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:34:38.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:38.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:38.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:38.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:38.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: fsmap 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mgrmap e13: x(active, since 51s) 2023-07-20T02:34:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:38.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:38.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/3658866804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:34:38.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:39.004 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.011+0000 7f80c3312700 1 -- 172.21.15.17:0/2838018245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc105180 msgr2=0x7f80bc105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:39.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.011+0000 7f80c3312700 1 --2- 172.21.15.17:0/2838018245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc105180 0x7f80bc105560 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009910 tx=0x7f80b8009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:39.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.011+0000 7f80c3312700 1 -- 172.21.15.17:0/2838018245 shutdown_connections 2023-07-20T02:34:39.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.011+0000 7f80c3312700 1 --2- 172.21.15.17:0/2838018245 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc105180 0x7f80bc105560 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:39.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.011+0000 7f80c3312700 1 -- 172.21.15.17:0/2838018245 >> 172.21.15.17:0/2838018245 conn(0x7f80bc074c70 msgr2=0x7f80bc075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:39.005 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.011+0000 7f80c3312700 1 -- 172.21.15.17:0/2838018245 shutdown_connections 2023-07-20T02:34:39.006 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.011+0000 7f80c3312700 1 -- 172.21.15.17:0/2838018245 wait complete. 2023-07-20T02:34:39.007 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 Processor -- start 2023-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 -- start start 2023-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80bc105180 0x7f80bc0fc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc0fcc40 0x7f80bc0fd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80bc1a1a40 0x7f80bc1a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc1a4480 con 0x7f80bc0fcc40 2023-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc1a45f0 con 0x7f80bc1a1a40 2023-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c3312700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc1a47b0 con 0x7f80bc105180 2023-07-20T02:34:39.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c08ad700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc0fcc40 0x7f80bc0fd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:39.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c08ad700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc0fcc40 0x7f80bc0fd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42688/0 (socket says 172.21.15.17:42688) 2023-07-20T02:34:39.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.012+0000 7f80c08ad700 1 -- 172.21.15.17:0/1927145606 learned_addr learned my addr 172.21.15.17:0/1927145606 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:34:39.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c10ae700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80bc105180 0x7f80bc0fc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:39.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c08ad700 1 -- 172.21.15.17:0/1927145606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80bc105180 msgr2=0x7f80bc0fc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:39.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c08ad700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80bc105180 0x7f80bc0fc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:39.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c18af700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80bc1a1a40 0x7f80bc1a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:39.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c08ad700 1 -- 172.21.15.17:0/1927145606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80bc1a1a40 msgr2=0x7f80bc1a3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:39.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c08ad700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80bc1a1a40 0x7f80bc1a3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:39.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c08ad700 1 -- 172.21.15.17:0/1927145606 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b80095f0 con 0x7f80bc0fcc40 2023-07-20T02:34:39.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c10ae700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80bc105180 0x7f80bc0fc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:34:39.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c08ad700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc0fcc40 0x7f80bc0fd0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f80b000b780 tx=0x7f80b000bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80ae7fc700 1 -- 172.21.15.17:0/1927145606 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0004230 con 0x7f80bc0fcc40 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80ae7fc700 1 -- 172.21.15.17:0/1927145606 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80b0004390 con 0x7f80bc0fcc40 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc1a4a90 con 0x7f80bc0fcc40 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80ae7fc700 1 -- 172.21.15.17:0/1927145606 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0011b30 con 0x7f80bc0fcc40 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.013+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc1a4fe0 con 0x7f80bc0fcc40 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.014+0000 7f80ae7fc700 1 -- 172.21.15.17:0/1927145606 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f80b0011cd0 con 0x7f80bc0fcc40 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.014+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc061da0 con 0x7f80bc0fcc40 2023-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.014+0000 7f80ae7fc700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80a8038c30 0x7f80a803b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:39.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.014+0000 7f80ae7fc700 1 -- 172.21.15.17:0/1927145606 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f80b004ae50 con 0x7f80bc0fcc40 2023-07-20T02:34:39.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.014+0000 7f80c10ae700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80a8038c30 0x7f80a803b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:39.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.015+0000 7f80c10ae700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80a8038c30 0x7f80a803b0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f80b8006010 tx=0x7f80b800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:39.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.019+0000 7f80ae7fc700 1 -- 172.21.15.17:0/1927145606 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f80b001ae40 con 0x7f80bc0fcc40 2023-07-20T02:34:39.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.b calling monitor election 2023-07-20T02:34:39.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.b calling monitor election 2023-07-20T02:34:39.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.c calling monitor election 2023-07-20T02:34:39.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.a calling monitor election 2023-07-20T02:34:39.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T02:34:39.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:39.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: fsmap 2023-07-20T02:34:39.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:39.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: mgrmap e13: x(active, since 53s) 2023-07-20T02:34:39.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T02:34:39.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: Cluster is now healthy 2023-07-20T02:34:39.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: overall HEALTH_OK 2023-07-20T02:34:39.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:39.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:39.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:39.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: mon.b calling monitor election 2023-07-20T02:34:39.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: mon.b calling monitor election 2023-07-20T02:34:39.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: mon.c calling monitor election 2023-07-20T02:34:39.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: mon.a calling monitor election 2023-07-20T02:34:39.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T02:34:39.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:39.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: fsmap 2023-07-20T02:34:39.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:39.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: mgrmap e13: x(active, since 53s) 2023-07-20T02:34:39.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T02:34:39.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: Cluster is now healthy 2023-07-20T02:34:39.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: overall HEALTH_OK 2023-07-20T02:34:39.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:39.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:39.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:39.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: mon.b calling monitor election 2023-07-20T02:34:39.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: mon.b calling monitor election 2023-07-20T02:34:39.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: mon.c calling monitor election 2023-07-20T02:34:39.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: mon.a calling monitor election 2023-07-20T02:34:39.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T02:34:39.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} 2023-07-20T02:34:39.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: fsmap 2023-07-20T02:34:39.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:34:39.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: mgrmap e13: x(active, since 53s) 2023-07-20T02:34:39.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T02:34:39.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: Cluster is now healthy 2023-07-20T02:34:39.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: overall HEALTH_OK 2023-07-20T02:34:39.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:39.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:39.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:39.218 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.225+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f80bc0fdc30 con 0x7f80bc0fcc40 2023-07-20T02:34:39.218 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.225+0000 7f80ae7fc700 1 -- 172.21.15.17:0/1927145606 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f80b004c030 con 0x7f80bc0fcc40 2023-07-20T02:34:39.219 INFO:teuthology.orchestra.run.smithi017.stdout:# minimal ceph.conf for 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:34:39.219 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2023-07-20T02:34:39.219 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:34:39.219 INFO:teuthology.orchestra.run.smithi017.stdout: mon_host = [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-07-20T02:34:39.220 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80a8038c30 msgr2=0x7f80a803b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:39.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80a8038c30 0x7f80a803b0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f80b8006010 tx=0x7f80b800b540 comp rx=0 tx=0).stop 2023-07-20T02:34:39.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc0fcc40 msgr2=0x7f80bc0fd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:39.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc0fcc40 0x7f80bc0fd0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f80b000b780 tx=0x7f80b000bb40 comp rx=0 tx=0).stop 2023-07-20T02:34:39.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 shutdown_connections 2023-07-20T02:34:39.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80a8038c30 0x7f80a803b0f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:39.221 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80bc1a1a40 0x7f80bc1a3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80bc105180 0x7f80bc0fc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 --2- 172.21.15.17:0/1927145606 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f80bc0fcc40 0x7f80bc0fd0c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 >> 172.21.15.17:0/1927145606 conn(0x7f80bc074c70 msgr2=0x7f80bc102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 shutdown_connections 2023-07-20T02:34:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:39.228+0000 7f80c3312700 1 -- 172.21.15.17:0/1927145606 wait complete. 2023-07-20T02:34:39.877 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-20T02:34:39.877 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:34:39.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T02:34:39.909 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:34:39.910 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:39.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:39 smithi017 ceph-mon[127345]: Updating smithi017:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:39.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:39 smithi017 ceph-mon[127345]: Updating smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:39.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:39 smithi017 ceph-mon[127345]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:39.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:39 smithi017 ceph-mon[127345]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T02:34:39.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:39 smithi017 ceph-mon[127345]: Reconfiguring daemon mon.a on smithi017 2023-07-20T02:34:39.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:39 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:39.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:39 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1927145606' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:39.979 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:34:39.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T02:34:40.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:39 smithi146 ceph-mon[128841]: Updating smithi017:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:40.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:39 smithi146 ceph-mon[128841]: Updating smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:40.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:39 smithi146 ceph-mon[128841]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:40.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:39 smithi146 ceph-mon[128841]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T02:34:40.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:39 smithi146 ceph-mon[128841]: Reconfiguring daemon mon.a on smithi017 2023-07-20T02:34:40.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:39 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:40.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:39 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1927145606' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:40.018 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:34:40.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:40.085 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:34:40.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T02:34:40.103 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:39 smithi151 ceph-mon[129163]: Updating smithi017:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:40.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:39 smithi151 ceph-mon[129163]: Updating smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:40.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:39 smithi151 ceph-mon[129163]: Updating smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:34:40.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:39 smithi151 ceph-mon[129163]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T02:34:40.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:39 smithi151 ceph-mon[129163]: Reconfiguring daemon mon.a on smithi017 2023-07-20T02:34:40.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:39 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:34:40.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:39 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1927145606' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:40.123 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:34:40.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:34:40.195 INFO:tasks.cephadm:Adding mgr.x on smithi017 2023-07-20T02:34:40.195 INFO:tasks.cephadm:Adding mgr.y on smithi146 2023-07-20T02:34:40.195 INFO:tasks.cephadm:Adding mgr.z on smithi151 2023-07-20T02:34:40.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch apply mgr '3;smithi017=x;smithi146=y;smithi151=z' 2023-07-20T02:34:40.786 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:34:40.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:40 smithi146 ceph-mon[128841]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:40.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:40.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:40.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:40.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:40.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:40.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:41.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:40 smithi151 ceph-mon[129163]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:41.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:41.039 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:41.039 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:41.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:40 smithi017 ceph-mon[127345]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:41.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:41.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:41.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:41.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:41 smithi151 ceph-mon[129163]: Reconfiguring mon.b (monmap changed)... 2023-07-20T02:34:41.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:41 smithi151 ceph-mon[129163]: Reconfiguring daemon mon.b on smithi146 2023-07-20T02:34:41.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.884 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:41.884 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:41.884 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:41.884 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:42.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:41 smithi146 ceph-mon[128841]: Reconfiguring mon.b (monmap changed)... 2023-07-20T02:34:42.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:41 smithi146 ceph-mon[128841]: Reconfiguring daemon mon.b on smithi146 2023-07-20T02:34:42.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:42.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:42.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:42.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:42.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 -- 172.21.15.151:0/1049599694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 msgr2=0x7f24080ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 --2- 172.21.15.151:0/1049599694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 0x7f24080ff6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009910 tx=0x7f23fc009c20 comp rx=0 tx=0).stop 2023-07-20T02:34:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 -- 172.21.15.151:0/1049599694 shutdown_connections 2023-07-20T02:34:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 --2- 172.21.15.151:0/1049599694 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 0x7f24080ff6b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 -- 172.21.15.151:0/1049599694 >> 172.21.15.151:0/1049599694 conn(0x7f24080fa810 msgr2=0x7f24080fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 -- 172.21.15.151:0/1049599694 shutdown_connections 2023-07-20T02:34:42.085 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 -- 172.21.15.151:0/1049599694 wait complete. 2023-07-20T02:34:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.092+0000 7f240dd02700 1 Processor -- start 2023-07-20T02:34:42.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f240dd02700 1 -- start start 2023-07-20T02:34:42.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f240dd02700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 0x7f24080692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:42.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f240dd02700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2408198330 0x7f240819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:42.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f240dd02700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f240819acb0 0x7f240819d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:42.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f240dd02700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240819d8f0 con 0x7f24080ff290 2023-07-20T02:34:42.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f240dd02700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240819da60 con 0x7f2408198330 2023-07-20T02:34:42.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f240dd02700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240819dc20 con 0x7f240819acb0 2023-07-20T02:34:42.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f240819acb0 0x7f240819d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f240819acb0 0x7f240819d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40722/0 (socket says 172.21.15.151:40722) 2023-07-20T02:34:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 -- 172.21.15.151:0/965224408 learned_addr learned my addr 172.21.15.151:0/965224408 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:34:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f24077fe700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 0x7f24080692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2406ffd700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2408198330 0x7f240819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 -- 172.21.15.151:0/965224408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2408198330 msgr2=0x7f240819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2408198330 0x7f240819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 -- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 msgr2=0x7f24080692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 0x7f24080692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 -- 172.21.15.151:0/965224408 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0095f0 con 0x7f240819acb0 2023-07-20T02:34:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.093+0000 7f2407fff700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f240819acb0 0x7f240819d150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f240000ba50 tx=0x7f240000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.094+0000 7f2404ff9700 1 -- 172.21.15.151:0/965224408 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240000c9b0 con 0x7f240819acb0 2023-07-20T02:34:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.094+0000 7f2404ff9700 1 -- 172.21.15.151:0/965224408 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2400014440 con 0x7f240819acb0 2023-07-20T02:34:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.094+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240819df00 con 0x7f240819acb0 2023-07-20T02:34:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.094+0000 7f2404ff9700 1 -- 172.21.15.151:0/965224408 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2400007590 con 0x7f240819acb0 2023-07-20T02:34:42.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.094+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240819e3f0 con 0x7f240819acb0 2023-07-20T02:34:42.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.095+0000 7f2404ff9700 1 -- 172.21.15.151:0/965224408 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f24000077b0 con 0x7f240819acb0 2023-07-20T02:34:42.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.095+0000 7f2404ff9700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f23f0038c30 0x7f23f003b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:42.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.095+0000 7f2404ff9700 1 -- 172.21.15.151:0/965224408 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f240004b990 con 0x7f240819acb0 2023-07-20T02:34:42.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.095+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240818dd50 con 0x7f240819acb0 2023-07-20T02:34:42.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.095+0000 7f24077fe700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f23f0038c30 0x7f23f003b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:42.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.096+0000 7f24077fe700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f23f0038c30 0x7f23f003b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23fc005130 tx=0x7f23fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:42.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.100+0000 7f2404ff9700 1 -- 172.21.15.151:0/965224408 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f240001b8b0 con 0x7f240819acb0 2023-07-20T02:34:42.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:41 smithi017 ceph-mon[127345]: Reconfiguring mon.b (monmap changed)... 2023-07-20T02:34:42.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:41 smithi017 ceph-mon[127345]: Reconfiguring daemon mon.b on smithi146 2023-07-20T02:34:42.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:42.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:42.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:34:42.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:34:42.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:42.291 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.298+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi017=x;smithi146=y;smithi151=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f2408060bb0 con 0x7f23f0038c30 2023-07-20T02:34:42.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.306+0000 7f2404ff9700 1 -- 172.21.15.151:0/965224408 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2408060bb0 con 0x7f23f0038c30 2023-07-20T02:34:42.298 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mgr update... 2023-07-20T02:34:42.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f23f0038c30 msgr2=0x7f23f003b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:42.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f23f0038c30 0x7f23f003b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23fc005130 tx=0x7f23fc005fd0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f240819acb0 msgr2=0x7f240819d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:42.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f240819acb0 0x7f240819d150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f240000ba50 tx=0x7f240000bd60 comp rx=0 tx=0).stop 2023-07-20T02:34:42.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 shutdown_connections 2023-07-20T02:34:42.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f23f0038c30 0x7f23f003b0f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f240819acb0 0x7f240819d150 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f24080ff290 0x7f24080692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 --2- 172.21.15.151:0/965224408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2408198330 0x7f240819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:42.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 >> 172.21.15.151:0/965224408 conn(0x7f24080fa810 msgr2=0x7f24080fb550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:42.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 shutdown_connections 2023-07-20T02:34:42.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:34:42.308+0000 7f240dd02700 1 -- 172.21.15.151:0/965224408 wait complete. 2023-07-20T02:34:42.748 DEBUG:teuthology.orchestra.run.smithi146:mgr.y> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.y.service 2023-07-20T02:34:42.751 DEBUG:teuthology.orchestra.run.smithi151:mgr.z> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.z.service 2023-07-20T02:34:42.752 INFO:tasks.cephadm:Deploying OSDs... 2023-07-20T02:34:42.753 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-07-20T02:34:42.753 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T02:34:42.774 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T02:34:42.774 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2023-07-20T02:34:42.780 INFO:journalctl@ceph.mgr.z.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:34:42.785 INFO:journalctl@ceph.mgr.y.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:34:42.834 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T02:34:42.835 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:42.835 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 369586 Links: 1 2023-07-20T02:34:42.835 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:42.835 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:42.835 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-07-20 02:33:55.688811954 +0000 2023-07-20T02:34:42.835 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-07-20 02:21:33.252351656 +0000 2023-07-20T02:34:42.836 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-07-20 02:21:33.252351656 +0000 2023-07-20T02:34:42.836 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-07-20T02:34:42.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T02:34:42.905 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-07-20T02:34:42.905 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-07-20T02:34:42.905 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000132773 s, 3.9 MB/s 2023-07-20T02:34:42.906 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T02:34:42.967 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2023-07-20T02:34:43.027 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 373354 Links: 1 2023-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-07-20 02:33:55.688811954 +0000 2023-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-07-20 02:21:33.779346387 +0000 2023-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-07-20 02:21:33.779346387 +0000 2023-07-20T02:34:43.029 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-07-20T02:34:43.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T02:34:43.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:43.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: Reconfiguring mon.c (monmap changed)... 2023-07-20T02:34:43.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: Reconfiguring daemon mon.c on smithi151 2023-07-20T02:34:43.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:43.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:43.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:43.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:43.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:34:43.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:43.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:43.094 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:43.094 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: Reconfiguring mon.c (monmap changed)... 2023-07-20T02:34:43.094 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: Reconfiguring daemon mon.c on smithi151 2023-07-20T02:34:43.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:43.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:43.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:43.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:43.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:34:43.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:43.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:43.096 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-07-20T02:34:43.097 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-07-20T02:34:43.097 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.8674e-05 s, 5.2 MB/s 2023-07-20T02:34:43.098 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T02:34:43.103 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:43.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: Reconfiguring mon.c (monmap changed)... 2023-07-20T02:34:43.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: Reconfiguring daemon mon.c on smithi151 2023-07-20T02:34:43.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:43.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:43.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:43.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:43.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:43.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:34:43.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:43.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:43.158 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2023-07-20T02:34:43.217 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T02:34:43.218 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:43.218 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 381434 Links: 1 2023-07-20T02:34:43.218 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:43.218 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:43.218 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-07-20 02:33:55.688811954 +0000 2023-07-20T02:34:43.218 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-07-20 02:21:34.301341169 +0000 2023-07-20T02:34:43.218 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-07-20 02:21:34.301341169 +0000 2023-07-20T02:34:43.219 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-07-20T02:34:43.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T02:34:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-07-20T02:34:43.286 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-07-20T02:34:43.286 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000120021 s, 4.3 MB/s 2023-07-20T02:34:43.288 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T02:34:43.346 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2023-07-20T02:34:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T02:34:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:43.405 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 376536 Links: 1 2023-07-20T02:34:43.405 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:43.405 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:43.405 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-07-20 02:33:55.688811954 +0000 2023-07-20T02:34:43.405 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-07-20 02:21:34.806336120 +0000 2023-07-20T02:34:43.405 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-07-20 02:21:34.806336120 +0000 2023-07-20T02:34:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-07-20T02:34:43.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T02:34:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-07-20T02:34:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-07-20T02:34:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.00011122 s, 4.6 MB/s 2023-07-20T02:34:43.472 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T02:34:43.531 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-07-20T02:34:43.532 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T02:34:43.552 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T02:34:43.553 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2023-07-20T02:34:43.612 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T02:34:43.612 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:43.612 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 381119 Links: 1 2023-07-20T02:34:43.612 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:43.613 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:43.613 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-07-20 02:34:30.420986423 +0000 2023-07-20T02:34:43.613 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-07-20 02:27:54.783195371 +0000 2023-07-20T02:34:43.613 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-07-20 02:27:54.783195371 +0000 2023-07-20T02:34:43.613 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-07-20T02:34:43.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T02:34:43.680 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-07-20T02:34:43.681 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-07-20T02:34:43.681 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00025764 s, 2.0 MB/s 2023-07-20T02:34:43.682 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T02:34:43.742 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2023-07-20T02:34:43.801 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T02:34:43.801 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:43.801 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 379358 Links: 1 2023-07-20T02:34:43.801 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:43.801 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:43.802 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-07-20 02:34:30.420986423 +0000 2023-07-20T02:34:43.802 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-07-20 02:27:55.342186511 +0000 2023-07-20T02:34:43.802 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-07-20 02:27:55.342186511 +0000 2023-07-20T02:34:43.802 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-07-20T02:34:43.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T02:34:43.869 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-07-20T02:34:43.869 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-07-20T02:34:43.870 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000292341 s, 1.8 MB/s 2023-07-20T02:34:43.871 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T02:34:43.931 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2023-07-20T02:34:43.989 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T02:34:43.989 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:43.990 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 371682 Links: 1 2023-07-20T02:34:43.990 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:43.990 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:43.990 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-07-20 02:34:30.421986408 +0000 2023-07-20T02:34:43.991 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-07-20 02:27:55.861178287 +0000 2023-07-20T02:34:43.991 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-07-20 02:27:55.861178287 +0000 2023-07-20T02:34:43.991 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-07-20T02:34:43.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T02:34:44.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:43 smithi146 ceph-mon[128841]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi017=x;smithi146=y;smithi151=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:44.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:43 smithi146 ceph-mon[128841]: Saving service mgr spec with placement smithi017=x;smithi146=y;smithi151=z;count:3 2023-07-20T02:34:44.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:43 smithi146 ceph-mon[128841]: Deploying daemon mgr.z on smithi151 2023-07-20T02:34:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-07-20T02:34:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-07-20T02:34:44.056 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000203364 s, 2.5 MB/s 2023-07-20T02:34:44.057 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T02:34:44.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:43 smithi151 ceph-mon[129163]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi017=x;smithi146=y;smithi151=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:44.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:43 smithi151 ceph-mon[129163]: Saving service mgr spec with placement smithi017=x;smithi146=y;smithi151=z;count:3 2023-07-20T02:34:44.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:43 smithi151 ceph-mon[129163]: Deploying daemon mgr.z on smithi151 2023-07-20T02:34:44.105 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:43 smithi151 systemd[1]: Starting Ceph mgr.z for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:34:44.115 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2023-07-20T02:34:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T02:34:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:44.174 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 377211 Links: 1 2023-07-20T02:34:44.174 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:44.174 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:44.174 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-07-20 02:34:30.421986408 +0000 2023-07-20T02:34:44.175 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-07-20 02:27:56.381170046 +0000 2023-07-20T02:34:44.175 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-07-20 02:27:56.381170046 +0000 2023-07-20T02:34:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-07-20T02:34:44.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T02:34:44.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:43 smithi017 ceph-mon[127345]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi017=x;smithi146=y;smithi151=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:44.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:43 smithi017 ceph-mon[127345]: Saving service mgr spec with placement smithi017=x;smithi146=y;smithi151=z;count:3 2023-07-20T02:34:44.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:43 smithi017 ceph-mon[127345]: Deploying daemon mgr.z on smithi151 2023-07-20T02:34:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-07-20T02:34:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-07-20T02:34:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000200259 s, 2.6 MB/s 2023-07-20T02:34:44.244 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T02:34:44.302 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-07-20T02:34:44.303 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T02:34:44.321 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T02:34:44.321 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2023-07-20T02:34:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T02:34:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:44.379 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 370183 Links: 1 2023-07-20T02:34:44.379 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:44.380 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:44.380 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-07-20 02:34:43.865376074 +0000 2023-07-20T02:34:44.380 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-07-20 02:21:41.206863124 +0000 2023-07-20T02:34:44.380 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-07-20 02:21:41.206863124 +0000 2023-07-20T02:34:44.380 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-07-20T02:34:44.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T02:34:44.448 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-07-20T02:34:44.448 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-07-20T02:34:44.448 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.00023771 s, 2.2 MB/s 2023-07-20T02:34:44.449 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T02:34:44.472 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 podman[131010]: 2023-07-20T02:34:44.488 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2023-07-20T02:34:44.548 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T02:34:44.548 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:44.548 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 374626 Links: 1 2023-07-20T02:34:44.548 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:44.548 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:44.549 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-07-20 02:34:43.865376074 +0000 2023-07-20T02:34:44.549 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-07-20 02:21:41.761856246 +0000 2023-07-20T02:34:44.549 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-07-20 02:21:41.761856246 +0000 2023-07-20T02:34:44.549 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-07-20T02:34:44.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T02:34:44.615 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-07-20T02:34:44.615 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-07-20T02:34:44.615 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000229348 s, 2.2 MB/s 2023-07-20T02:34:44.616 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T02:34:44.673 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2023-07-20T02:34:44.730 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T02:34:44.730 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:44.730 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 369454 Links: 1 2023-07-20T02:34:44.731 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:44.731 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:44.731 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-07-20 02:34:43.865376074 +0000 2023-07-20T02:34:44.731 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-07-20 02:21:42.267849977 +0000 2023-07-20T02:34:44.731 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-07-20 02:21:42.267849977 +0000 2023-07-20T02:34:44.731 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-07-20T02:34:44.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T02:34:44.740 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 bash[131010]: 3f02d844eae6bca0978542c86736d598f2130c613d1a35652b6f6de2e693d04d 2023-07-20T02:34:44.740 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mgr[131108]: -- 172.21.15.151:0/2612467179 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5581db66b1e0 con 0x5581db641000 2023-07-20T02:34:44.740 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 systemd[1]: Started Ceph mgr.z for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:34:44.741 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:44.606+0000 7f260a698200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:34:44.741 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:44.690+0000 7f260a698200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:34:44.766 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-07-20T02:34:44.766 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-07-20T02:34:44.766 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000217003 s, 2.4 MB/s 2023-07-20T02:34:44.767 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T02:34:44.824 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2023-07-20T02:34:44.881 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T02:34:44.881 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:34:44.881 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 379628 Links: 1 2023-07-20T02:34:44.882 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:34:44.882 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:34:44.882 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-07-20 02:34:43.865376074 +0000 2023-07-20T02:34:44.882 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-07-20 02:21:42.772843719 +0000 2023-07-20T02:34:44.882 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-07-20 02:21:42.772843719 +0000 2023-07-20T02:34:44.883 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-07-20T02:34:44.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T02:34:44.947 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-07-20T02:34:44.947 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-07-20T02:34:44.948 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000219635 s, 2.3 MB/s 2023-07-20T02:34:44.949 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T02:34:45.006 INFO:tasks.cephadm:Deploying osd.0 on smithi017 with /dev/vg_nvme/lv_4... 2023-07-20T02:34:45.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T02:34:45.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:45.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.038 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.038 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:45.038 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:34:45.038 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:45.038 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:45.054 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:44.785+0000 7f260a698200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:34:45.054 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:44.883+0000 7f260a698200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:34:45.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:45.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.056 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:45.056 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:34:45.056 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:45.056 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:45.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:45.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:45.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:45.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:34:45.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:45.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:45.354 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:45.061+0000 7f260a698200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:34:45.578 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:45.674 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:45.370+0000 7f260a698200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:34:45.674 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:45.459+0000 7f260a698200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:34:45.675 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:45.587+0000 7f260a698200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:34:45.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:45 smithi146 ceph-mon[128841]: Deploying daemon mgr.y on smithi146 2023-07-20T02:34:46.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:45 smithi017 ceph-mon[127345]: Deploying daemon mgr.y on smithi146 2023-07-20T02:34:46.095 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:45 smithi151 ceph-mon[129163]: Deploying daemon mgr.y on smithi146 2023-07-20T02:34:46.095 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:45.681+0000 7f260a698200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:34:46.354 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:46.102+0000 7f260a698200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:34:46.746 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:46 smithi146 systemd[1]: Starting Ceph mgr.y for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:34:46.747 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:46 smithi146 podman[130276]: 2023-07-20T02:34:46.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:46 smithi017 ceph-mon[127345]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:47.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:46 smithi146 ceph-mon[128841]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:47.053 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:46 smithi146 bash[130276]: 772d4ebc033370a0539d075e65f79565540738c87716de84f494b3d7889e3ebc 2023-07-20T02:34:47.053 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:46 smithi146 systemd[1]: Started Ceph mgr.y for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:34:47.053 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:46.882+0000 7fde9e8ec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:34:47.053 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:46.964+0000 7fde9e8ec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:34:47.054 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:47.059+0000 7fde9e8ec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:34:47.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:46 smithi151 ceph-mon[129163]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:47.104 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:46.678+0000 7f260a698200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:34:47.104 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:46.850+0000 7f260a698200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:34:47.177 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T02:34:47.177 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T02:34:47.177 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2023-07-20T02:34:47.178 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2023-07-20T02:34:47.178 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00826026 s, 1.3 GB/s 2023-07-20T02:34:47.178 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2023-07-20T02:34:47.310 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:47.161+0000 7fde9e8ec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:34:47.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi017:vg_nvme/lv_4 2023-07-20T02:34:47.679 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:47.350+0000 7fde9e8ec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:34:47.917 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:47.648+0000 7f260a698200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:34:47.917 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:47.746+0000 7f260a698200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:34:47.917 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:47.918 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:47.918 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-mon[129163]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:47.918 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:47.918 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:47.918 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:48.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-mon[128841]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:48.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:48.026 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:47.685+0000 7fde9e8ec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:34:48.026 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:47.780+0000 7fde9e8ec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:34:48.026 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:47.930+0000 7fde9e8ec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:34:48.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:47 smithi017 ceph-mon[127345]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:48.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:48.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:48.216 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:47.924+0000 7f260a698200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:34:48.240 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:34:48.604 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:48.223+0000 7f260a698200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:34:48.604 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:48.464+0000 7f260a698200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:34:48.810 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:48 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:48.031+0000 7fde9e8ec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:34:48.810 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:48 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:48.474+0000 7fde9e8ec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:34:48.878 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:48.618+0000 7f260a698200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:34:49.354 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:48.885+0000 7f260a698200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:34:49.499 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 -- 172.21.15.17:0/2501586160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb700ff730 msgr2=0x7fbb700ffb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:49.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 --2- 172.21.15.17:0/2501586160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb700ff730 0x7fbb700ffb50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0099c0 tx=0x7fbb6c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:34:49.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 -- 172.21.15.17:0/2501586160 shutdown_connections 2023-07-20T02:34:49.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 --2- 172.21.15.17:0/2501586160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb701012d0 0x7fbb70107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:49.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 --2- 172.21.15.17:0/2501586160 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb700ff730 0x7fbb700ffb50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:49.500 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 --2- 172.21.15.17:0/2501586160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb70100930 0x7fbb70100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:49.501 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 -- 172.21.15.17:0/2501586160 >> 172.21.15.17:0/2501586160 conn(0x7fbb700facf0 msgr2=0x7fbb700fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:34:49.501 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 -- 172.21.15.17:0/2501586160 shutdown_connections 2023-07-20T02:34:49.501 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.506+0000 7fbb78394700 1 -- 172.21.15.17:0/2501586160 wait complete. 2023-07-20T02:34:49.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 Processor -- start 2023-07-20T02:34:49.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 -- start start 2023-07-20T02:34:49.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb700ff730 0x7fbb70069750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:49.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb70100930 0x7fbb7019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:49.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb701012d0 0x7fbb7019d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:49.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7019dfd0 con 0x7fbb70100930 2023-07-20T02:34:49.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7019e140 con 0x7fbb700ff730 2023-07-20T02:34:49.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb78394700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7019e300 con 0x7fbb701012d0 2023-07-20T02:34:49.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb7592f700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb70100930 0x7fbb7019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:49.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb7592f700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb70100930 0x7fbb7019b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42702/0 (socket says 172.21.15.17:42702) 2023-07-20T02:34:49.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb7592f700 1 -- 172.21.15.17:0/3038932715 learned_addr learned my addr 172.21.15.17:0/3038932715 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:34:49.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb76130700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb700ff730 0x7fbb70069750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:49.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb76931700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb701012d0 0x7fbb7019d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:49.504 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.507+0000 7fbb7592f700 1 -- 172.21.15.17:0/3038932715 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb701012d0 msgr2=0x7fbb7019d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:49.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb7592f700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb701012d0 0x7fbb7019d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:49.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb7592f700 1 -- 172.21.15.17:0/3038932715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb700ff730 msgr2=0x7fbb70069750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:34:49.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb7592f700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb700ff730 0x7fbb70069750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:34:49.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb7592f700 1 -- 172.21.15.17:0/3038932715 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c0095f0 con 0x7fbb70100930 2023-07-20T02:34:49.505 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb76130700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb700ff730 0x7fbb70069750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:34:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb7592f700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb70100930 0x7fbb7019b1e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000e9a0 tx=0x7fbb6000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60009790 con 0x7fbb70100930 2023-07-20T02:34:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb60010430 con 0x7fbb70100930 2023-07-20T02:34:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7019e5e0 con 0x7fbb70100930 2023-07-20T02:34:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60018680 con 0x7fbb70100930 2023-07-20T02:34:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.508+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7019eb30 con 0x7fbb70100930 2023-07-20T02:34:49.506 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.509+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb70106cd0 con 0x7fbb70100930 2023-07-20T02:34:49.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.510+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fbb600105a0 con 0x7fbb70100930 2023-07-20T02:34:49.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.510+0000 7fbb677fe700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbb5c0389e0 0x7fbb5c03aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:34:49.507 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.510+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbb60014070 con 0x7fbb70100930 2023-07-20T02:34:49.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.510+0000 7fbb76130700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbb5c0389e0 0x7fbb5c03aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:34:49.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.510+0000 7fbb76130700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbb5c0389e0 0x7fbb5c03aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009990 tx=0x7fbb6c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:34:49.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.516+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbb6001f780 con 0x7fbb70100930 2023-07-20T02:34:49.560 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:49.071+0000 7fde9e8ec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:34:49.560 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:49.242+0000 7fde9e8ec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:34:49.734 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:49.740+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb70060bb0 con 0x7fbb5c0389e0 2023-07-20T02:34:49.854 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:49.597+0000 7f260a698200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:34:49.855 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:49.728+0000 7f260a698200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:34:49.855 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.855 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:49.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:49.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:49.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:49.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:49.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:49.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:49.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:49.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:49.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:49.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:50.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:50.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:50.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:34:50.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:34:50.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:50.247 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:49.923+0000 7f260a698200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:34:50.247 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:50.037+0000 7f260a698200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:34:50.271 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:50.012+0000 7fde9e8ec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:34:50.271 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:50.104+0000 7fde9e8ec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:34:50.547 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:50.276+0000 7fde9e8ec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:34:50.810 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:50.553+0000 7fde9e8ec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:34:50.811 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:50.785+0000 7fde9e8ec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:34:50.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:50.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T02:34:50.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: Reconfiguring daemon mgr.x on smithi017 2023-07-20T02:34:50.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:50.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:34:50.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:34:50.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:50.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:50.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:50.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:50.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.820 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:50.820 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T02:34:50.820 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: Reconfiguring daemon mgr.x on smithi017 2023-07-20T02:34:50.820 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:50.820 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:34:50.821 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:34:50.821 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:50.821 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.821 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.821 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:50.821 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:50.821 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:50.822 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:50.822 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:51.104 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:50.254+0000 7f260a698200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:34:51.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:51.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T02:34:51.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: Reconfiguring daemon mgr.x on smithi017 2023-07-20T02:34:51.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:34:51.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:34:51.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:34:51.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:51.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:51.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:51.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:51.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:34:51.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:51.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:34:51.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:34:51.187 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:50.934+0000 7fde9e8ec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:34:51.560 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:51 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:51.193+0000 7fde9e8ec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:34:51.604 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:34:51 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z[131025]: 2023-07-20T02:34:51.120+0000 7f260a698200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:34:51.735 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:51.740+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7fbb600187e0 con 0x7fbb70100930 2023-07-20T02:34:52.026 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:51 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:51.902+0000 7fde9e8ec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:34:52.027 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:51 smithi146 ceph-mon[128841]: Standby manager daemon z started 2023-07-20T02:34:52.027 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:51 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T02:34:52.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:51 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:34:52.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:51 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T02:34:52.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:51 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:34:52.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:51 smithi151 ceph-mon[129163]: Standby manager daemon z started 2023-07-20T02:34:52.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:51 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T02:34:52.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:51 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:34:52.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:51 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T02:34:52.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:51 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:34:52.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:51 smithi017 ceph-mon[127345]: Standby manager daemon z started 2023-07-20T02:34:52.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:51 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T02:34:52.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:51 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:34:52.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:51 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T02:34:52.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:51 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.151:0/2550118094' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:34:52.310 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:52.032+0000 7fde9e8ec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:34:52.310 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:52.224+0000 7fde9e8ec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:34:52.731 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:52.338+0000 7fde9e8ec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:34:52.731 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:52.554+0000 7fde9e8ec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:34:53.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:53.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-20T02:34:53.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T02:34:53.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3838969713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb89ee2e-e4ff-4c83-9626-69cbfb526dd7"}]: dispatch 2023-07-20T02:34:53.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3838969713' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb89ee2e-e4ff-4c83-9626-69cbfb526dd7"}]': finished 2023-07-20T02:34:53.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T02:34:53.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:34:53.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:52 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/269870706' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:34:53.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:53.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-20T02:34:53.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T02:34:53.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3838969713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb89ee2e-e4ff-4c83-9626-69cbfb526dd7"}]: dispatch 2023-07-20T02:34:53.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3838969713' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb89ee2e-e4ff-4c83-9626-69cbfb526dd7"}]': finished 2023-07-20T02:34:53.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T02:34:53.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:34:53.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:52 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/269870706' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:34:53.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:53.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-20T02:34:53.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T02:34:53.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3838969713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb89ee2e-e4ff-4c83-9626-69cbfb526dd7"}]: dispatch 2023-07-20T02:34:53.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3838969713' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb89ee2e-e4ff-4c83-9626-69cbfb526dd7"}]': finished 2023-07-20T02:34:53.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T02:34:53.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:34:53.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:52 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/269870706' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:34:53.737 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:34:53 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y[130291]: 2023-07-20T02:34:53.415+0000 7fde9e8ec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:34:53.741 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:34:53.748+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mon.0 v2:172.21.15.17:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fbb6007d480 con 0x7fbb70100930 2023-07-20T02:34:54.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:53 smithi146 ceph-mon[128841]: Standby manager daemon y started 2023-07-20T02:34:54.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:53 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T02:34:54.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:53 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:34:54.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:53 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T02:34:54.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:53 smithi146 ceph-mon[128841]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:34:54.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:53 smithi151 ceph-mon[129163]: Standby manager daemon y started 2023-07-20T02:34:54.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:53 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T02:34:54.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:53 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:34:54.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:53 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T02:34:54.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:53 smithi151 ceph-mon[129163]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:34:54.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:53 smithi017 ceph-mon[127345]: Standby manager daemon y started 2023-07-20T02:34:54.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:53 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T02:34:54.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:53 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:34:54.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:53 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T02:34:54.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:53 smithi017 ceph-mon[127345]: from='mgr.? 172.21.15.146:0/759434155' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:34:55.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:54 smithi146 ceph-mon[128841]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:55.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:54 smithi146 ceph-mon[128841]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-20T02:34:55.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T02:34:55.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:54 smithi151 ceph-mon[129163]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:55.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:54 smithi151 ceph-mon[129163]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-20T02:34:55.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T02:34:55.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:54 smithi017 ceph-mon[127345]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:55.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:54 smithi017 ceph-mon[127345]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-20T02:34:55.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T02:34:57.044 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:56 smithi017 ceph-mon[127345]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:57.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:56 smithi146 ceph-mon[128841]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:57.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:56 smithi151 ceph-mon[129163]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:58.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:58 smithi017 ceph-mon[127345]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:58.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T02:34:58.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:59.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:58 smithi146 ceph-mon[128841]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:59.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T02:34:59.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:34:59.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:58 smithi151 ceph-mon[129163]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:34:59.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T02:34:59.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:00.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:34:59 smithi146 ceph-mon[128841]: Deploying daemon osd.0 on smithi017 2023-07-20T02:35:00.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:34:59 smithi017 ceph-mon[127345]: Deploying daemon osd.0 on smithi017 2023-07-20T02:35:00.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:34:59 smithi151 ceph-mon[129163]: Deploying daemon osd.0 on smithi017 2023-07-20T02:35:01.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:00 smithi146 ceph-mon[128841]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:01.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:00 smithi151 ceph-mon[129163]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:01.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:00 smithi017 ceph-mon[127345]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:02.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:02 smithi017 ceph-mon[127345]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:02.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:02 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:02.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:02 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:02.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:02 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:03.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:02 smithi146 ceph-mon[128841]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:03.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:02 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:03.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:02 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:03.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:02 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:03.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:02 smithi151 ceph-mon[129163]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:03.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:02 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:03.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:02 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:03.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:02 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:05.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:04 smithi146 ceph-mon[128841]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:05.094 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:04 smithi017 ceph-mon[127345]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:05.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:04 smithi151 ceph-mon[129163]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:05.336 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.343+0000 7fbb677fe700 1 -- 172.21.15.17:0/3038932715 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbb70060bb0 con 0x7fbb5c0389e0 2023-07-20T02:35:05.337 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 0 on host 'smithi017' 2023-07-20T02:35:05.338 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbb5c0389e0 msgr2=0x7fbb5c03aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:05.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbb5c0389e0 0x7fbb5c03aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009990 tx=0x7fbb6c01a040 comp rx=0 tx=0).stop 2023-07-20T02:35:05.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb70100930 msgr2=0x7fbb7019b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:05.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb70100930 0x7fbb7019b1e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000e9a0 tx=0x7fbb6000ecb0 comp rx=0 tx=0).stop 2023-07-20T02:35:05.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 shutdown_connections 2023-07-20T02:35:05.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbb5c0389e0 0x7fbb5c03aea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:05.339 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb701012d0 0x7fbb7019d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:05.340 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb700ff730 0x7fbb70069750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:05.340 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 --2- 172.21.15.17:0/3038932715 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbb70100930 0x7fbb7019b1e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:05.340 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 >> 172.21.15.17:0/3038932715 conn(0x7fbb700facf0 msgr2=0x7fbb70103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:35:05.340 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 shutdown_connections 2023-07-20T02:35:05.340 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:05.346+0000 7fbb78394700 1 -- 172.21.15.17:0/3038932715 wait complete. 2023-07-20T02:35:05.745 DEBUG:teuthology.orchestra.run.smithi017:osd.0> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.0.service 2023-07-20T02:35:05.747 INFO:tasks.cephadm:Deploying osd.1 on smithi146 with /dev/vg_nvme/lv_4... 2023-07-20T02:35:05.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T02:35:05.785 INFO:journalctl@ceph.osd.0.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:35:05.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:05.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:05.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:05.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:05.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:05.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:05.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:05.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:06.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:06.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:06.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:06.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:06.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:06.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:06.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:06.310 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:35:06.785 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:06 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-0[140347]: 2023-07-20T02:35:06.536+0000 7fc4a8755640 -1 osd.0 0 log_to_monitors true 2023-07-20T02:35:06.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:06 smithi146 ceph-mon[128841]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:06.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:06 smithi146 ceph-mon[128841]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:35:07.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:06 smithi017 ceph-mon[127345]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:07.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:06 smithi017 ceph-mon[127345]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:35:07.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:06 smithi151 ceph-mon[129163]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:07.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:06 smithi151 ceph-mon[129163]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:35:07.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:07 smithi146 ceph-mon[128841]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T02:35:07.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:07 smithi146 ceph-mon[128841]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T02:35:07.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:07 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:07.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:07 smithi146 ceph-mon[128841]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:35:07.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:07 smithi146 ceph-mon[128841]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:35:07.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:07 smithi146 ceph-mon[128841]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T02:35:07.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:07 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:07.947 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-0[140347]: 2023-07-20T02:35:07.795+0000 7fc49d5cc700 -1 osd.0 0 waiting for initial osdmap 2023-07-20T02:35:07.948 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-0[140347]: 2023-07-20T02:35:07.798+0000 7fc498bc1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:35:07.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-mon[127345]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T02:35:07.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-mon[127345]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T02:35:07.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:07.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-mon[127345]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:35:07.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-mon[127345]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:35:07.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-mon[127345]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T02:35:07.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:07 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:08.074 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T02:35:08.075 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T02:35:08.075 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2023-07-20T02:35:08.075 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2023-07-20T02:35:08.075 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111151 s, 943 MB/s 2023-07-20T02:35:08.076 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2023-07-20T02:35:08.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:07 smithi151 ceph-mon[129163]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T02:35:08.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:07 smithi151 ceph-mon[129163]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T02:35:08.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:07 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:08.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:07 smithi151 ceph-mon[129163]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:35:08.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:07 smithi151 ceph-mon[129163]: from='osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:35:08.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:07 smithi151 ceph-mon[129163]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T02:35:08.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:07 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:08.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2023-07-20T02:35:08.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:08.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:08.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:08.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:08.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:08.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:08.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:08.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:08.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:08.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:09.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:09.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:09.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:09.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:09.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:09.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:09.122 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:35:09.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:09.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:09.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:09.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:09.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:09.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:09.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:09.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:35:09.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:35:09.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi017 2023-07-20T02:35:09.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi017 to 17173M 2023-07-20T02:35:09.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069] boot 2023-07-20T02:35:09.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T02:35:09.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:09.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:09 smithi146 ceph-mon[128841]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:10.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:35:10.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:35:10.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi017 2023-07-20T02:35:10.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi017 to 17173M 2023-07-20T02:35:10.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069] boot 2023-07-20T02:35:10.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T02:35:10.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:10.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:09 smithi151 ceph-mon[129163]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:10.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:35:10.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:35:10.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi017 2023-07-20T02:35:10.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi017 to 17173M 2023-07-20T02:35:10.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: osd.0 [v2:172.21.15.17:6802/795351069,v1:172.21.15.17:6803/795351069] boot 2023-07-20T02:35:10.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T02:35:10.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:35:10.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:09 smithi017 ceph-mon[127345]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:35:10.440 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 -- 172.21.15.146:0/3687046701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f4106550 msgr2=0x7f28f41089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:10.441 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 --2- 172.21.15.146:0/3687046701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f4106550 0x7f28f41089c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00b0a0 tx=0x7f28ec00b3b0 comp rx=0 tx=0).stop 2023-07-20T02:35:10.442 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 -- 172.21.15.146:0/3687046701 shutdown_connections 2023-07-20T02:35:10.442 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 --2- 172.21.15.146:0/3687046701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f4106550 0x7f28f41089c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:10.442 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 --2- 172.21.15.146:0/3687046701 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28f41012f0 0x7f28f41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 --2- 172.21.15.146:0/3687046701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28f4103c20 0x7f28f4106010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 -- 172.21.15.146:0/3687046701 >> 172.21.15.146:0/3687046701 conn(0x7f28f40fad00 msgr2=0x7f28f40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:35:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 -- 172.21.15.146:0/3687046701 shutdown_connections 2023-07-20T02:35:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.447+0000 7f28fc24c700 1 -- 172.21.15.146:0/3687046701 wait complete. 2023-07-20T02:35:10.446 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 Processor -- start 2023-07-20T02:35:10.446 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 -- start start 2023-07-20T02:35:10.446 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f41012f0 0x7f28f41943a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:10.446 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28f4103c20 0x7f28f41948e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:10.446 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28f4106550 0x7f28f419b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:10.446 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f419bf30 con 0x7f28f4106550 2023-07-20T02:35:10.446 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f419c070 con 0x7f28f41012f0 2023-07-20T02:35:10.447 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fc24c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f419c200 con 0x7f28f4103c20 2023-07-20T02:35:10.447 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28f9fe8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f41012f0 0x7f28f41943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:10.447 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28f9fe8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f41012f0 0x7f28f41943a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:52136/0 (socket says 172.21.15.146:52136) 2023-07-20T02:35:10.447 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28f9fe8700 1 -- 172.21.15.146:0/2508265388 learned_addr learned my addr 172.21.15.146:0/2508265388 (peer_addr_for_me v2:172.21.15.146:0/0) 2023-07-20T02:35:10.447 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28f97e7700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28f4103c20 0x7f28f41948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:10.447 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.448+0000 7f28fa7e9700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28f4106550 0x7f28f419b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:10.448 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28f9fe8700 1 -- 172.21.15.146:0/2508265388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28f4103c20 msgr2=0x7f28f41948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:10.448 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28f9fe8700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28f4103c20 0x7f28f41948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:10.448 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28f9fe8700 1 -- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28f4106550 msgr2=0x7f28f419b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:10.448 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28f9fe8700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28f4106550 0x7f28f419b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:10.449 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28f9fe8700 1 -- 172.21.15.146:0/2508265388 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec009d00 con 0x7f28f41012f0 2023-07-20T02:35:10.449 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28f9fe8700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f41012f0 0x7f28f41943a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28e4011b60 tx=0x7f28e4011e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:35:10.449 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28eaffd700 1 -- 172.21.15.146:0/2508265388 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e400f950 con 0x7f28f41012f0 2023-07-20T02:35:10.449 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28eaffd700 1 -- 172.21.15.146:0/2508265388 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28e4013040 con 0x7f28f41012f0 2023-07-20T02:35:10.449 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28eaffd700 1 -- 172.21.15.146:0/2508265388 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e40183a0 con 0x7f28f41012f0 2023-07-20T02:35:10.450 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f419c4e0 con 0x7f28f41012f0 2023-07-20T02:35:10.451 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.449+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f419c9e0 con 0x7f28f41012f0 2023-07-20T02:35:10.451 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.450+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f40fc880 con 0x7f28f41012f0 2023-07-20T02:35:10.451 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.451+0000 7f28eaffd700 1 -- 172.21.15.146:0/2508265388 <== mon.2 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f28e4019590 con 0x7f28f41012f0 2023-07-20T02:35:10.451 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.451+0000 7f28eaffd700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f28e00a11c0 0x7f28e00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:10.451 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.452+0000 7f28eaffd700 1 -- 172.21.15.146:0/2508265388 <== mon.2 v2:172.21.15.146:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f28e40ce450 con 0x7f28f41012f0 2023-07-20T02:35:10.451 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.452+0000 7f28f97e7700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f28e00a11c0 0x7f28e00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:10.451 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.452+0000 7f28f97e7700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f28e00a11c0 0x7f28e00a3680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f28f0005950 tx=0x7f28f000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:35:10.452 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.456+0000 7f28eaffd700 1 -- 172.21.15.146:0/2508265388 <== mon.2 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f28e4099570 con 0x7f28f41012f0 2023-07-20T02:35:10.641 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:10.647+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f28f4060bb0 con 0x7f28e00a11c0 2023-07-20T02:35:11.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:11 smithi146 ceph-mon[128841]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T02:35:11.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:11 smithi146 ceph-mon[128841]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:11.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:11.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:11.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:11.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:11 smithi151 ceph-mon[129163]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T02:35:11.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:11 smithi151 ceph-mon[129163]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:11.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:11.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:11.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:11.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:11 smithi017 ceph-mon[127345]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T02:35:11.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:11 smithi017 ceph-mon[127345]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:11.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:11.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:11.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:12.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:12 smithi146 ceph-mon[128841]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:12.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:12 smithi151 ceph-mon[129163]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:12.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:12 smithi017 ceph-mon[127345]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:13.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:13 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/822220490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]: dispatch 2023-07-20T02:35:13.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:13 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]: dispatch 2023-07-20T02:35:13.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:13 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]': finished 2023-07-20T02:35:13.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:13 smithi151 ceph-mon[129163]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T02:35:13.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:13.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:13 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/822220490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]: dispatch 2023-07-20T02:35:13.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:13 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]: dispatch 2023-07-20T02:35:13.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:13 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]': finished 2023-07-20T02:35:13.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:13 smithi017 ceph-mon[127345]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T02:35:13.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:13.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:13 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/822220490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]: dispatch 2023-07-20T02:35:13.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:13 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]: dispatch 2023-07-20T02:35:13.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:13 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d422389-0925-410c-bd28-50ad06d68cbf"}]': finished 2023-07-20T02:35:13.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:13 smithi146 ceph-mon[128841]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T02:35:13.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:14.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:14 smithi151 ceph-mon[129163]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:14.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:14 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/908888580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:14.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:14 smithi017 ceph-mon[127345]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:14.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:14 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/908888580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:14.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:14 smithi146 ceph-mon[128841]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:14.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:14 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/908888580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:16.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:16 smithi151 ceph-mon[129163]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:16.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:16 smithi017 ceph-mon[127345]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:16.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:16 smithi146 ceph-mon[128841]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:18.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:18 smithi146 ceph-mon[128841]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:18.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:18 smithi151 ceph-mon[129163]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:18.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:18 smithi017 ceph-mon[127345]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:20.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:20 smithi151 ceph-mon[129163]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:20.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:20 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T02:35:20.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:20 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:20.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:20 smithi146 ceph-mon[128841]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:20.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:20 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T02:35:20.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:20 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:20.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:20 smithi017 ceph-mon[127345]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:20.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:20 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T02:35:20.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:20 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:21.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:21 smithi151 ceph-mon[129163]: Deploying daemon osd.1 on smithi146 2023-07-20T02:35:21.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:21 smithi146 ceph-mon[128841]: Deploying daemon osd.1 on smithi146 2023-07-20T02:35:21.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:21 smithi017 ceph-mon[127345]: Deploying daemon osd.1 on smithi146 2023-07-20T02:35:22.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:22 smithi151 ceph-mon[129163]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:22.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:22 smithi017 ceph-mon[127345]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:22.700 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:22 smithi146 ceph-mon[128841]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:24.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:24 smithi151 ceph-mon[129163]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:24.609 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:24 smithi146 ceph-mon[128841]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:24.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:24 smithi017 ceph-mon[127345]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:25.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:25.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:25.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:25.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:25 smithi146 ceph-mon[128841]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:26.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:26.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:26.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:26.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:25 smithi151 ceph-mon[129163]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:26.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:26.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:26.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:26.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:25 smithi017 ceph-mon[127345]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:28.016 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.021+0000 7f28eaffd700 1 -- 172.21.15.146:0/2508265388 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f28f4060bb0 con 0x7f28e00a11c0 2023-07-20T02:35:28.016 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 1 on host 'smithi146' 2023-07-20T02:35:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f28e00a11c0 msgr2=0x7f28e00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f28e00a11c0 0x7f28e00a3680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f28f0005950 tx=0x7f28f000b410 comp rx=0 tx=0).stop 2023-07-20T02:35:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f41012f0 msgr2=0x7f28f41943a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f41012f0 0x7f28f41943a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28e4011b60 tx=0x7f28e4011e70 comp rx=0 tx=0).stop 2023-07-20T02:35:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 shutdown_connections 2023-07-20T02:35:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f28e00a11c0 0x7f28e00a3680 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f28f4106550 0x7f28f419b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28f41012f0 0x7f28f41943a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 --2- 172.21.15.146:0/2508265388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f28f4103c20 0x7f28f41948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 >> 172.21.15.146:0/2508265388 conn(0x7f28f40fad00 msgr2=0x7f28f4101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:35:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 shutdown_connections 2023-07-20T02:35:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:35:28.024+0000 7f28fc24c700 1 -- 172.21.15.146:0/2508265388 wait complete. 2023-07-20T02:35:28.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:28.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:28.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:28.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:28.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:28.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:28.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:28.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:28.505 DEBUG:teuthology.orchestra.run.smithi146:osd.1> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.1.service 2023-07-20T02:35:28.507 INFO:tasks.cephadm:Deploying osd.2 on smithi151 with /dev/vg_nvme/lv_4... 2023-07-20T02:35:28.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T02:35:28.542 INFO:journalctl@ceph.osd.1.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:35:28.542 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:28.542 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.542 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.542 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:28.542 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:28.543 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.543 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.543 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:28.543 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:29.037 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:28 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-1[133811]: 2023-07-20T02:35:28.779+0000 7fa8e9e61640 -1 osd.1 0 log_to_monitors true 2023-07-20T02:35:29.069 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:35:29.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:29 smithi146 ceph-mon[128841]: from='osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:35:29.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:29 smithi146 ceph-mon[128841]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:35:29.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:29 smithi017 ceph-mon[127345]: from='osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:35:29.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:29 smithi017 ceph-mon[127345]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:35:29.450 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:29 smithi151 ceph-mon[129163]: from='osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:35:29.451 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:29 smithi151 ceph-mon[129163]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:35:30.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:30 smithi017 ceph-mon[127345]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:30.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:30 smithi017 ceph-mon[127345]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T02:35:30.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:30 smithi017 ceph-mon[127345]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T02:35:30.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:30.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:30 smithi017 ceph-mon[127345]: from='osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:35:30.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:30 smithi017 ceph-mon[127345]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:35:30.496 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-1[133811]: 2023-07-20T02:35:30.182+0000 7fa8decd8700 -1 osd.1 0 waiting for initial osdmap 2023-07-20T02:35:30.496 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-1[133811]: 2023-07-20T02:35:30.186+0000 7fa8d8aca700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:35:30.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-mon[128841]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:30.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-mon[128841]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T02:35:30.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-mon[128841]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T02:35:30.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:30.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-mon[128841]: from='osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:35:30.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:30 smithi146 ceph-mon[128841]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:35:30.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:30 smithi151 ceph-mon[129163]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:30.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:30 smithi151 ceph-mon[129163]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T02:35:30.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:30 smithi151 ceph-mon[129163]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T02:35:30.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:30.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:30 smithi151 ceph-mon[129163]: from='osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:35:30.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:30 smithi151 ceph-mon[129163]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:35:30.621 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T02:35:30.622 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T02:35:30.622 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2023-07-20T02:35:30.622 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2023-07-20T02:35:30.622 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110949 s, 945 MB/s 2023-07-20T02:35:30.622 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2023-07-20T02:35:31.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi151:vg_nvme/lv_4 2023-07-20T02:35:31.355 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:35:31.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T02:35:31.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:31.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:31.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:31.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:31.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:31.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:35:31.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T02:35:31.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:31.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:35:31.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T02:35:31.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:31.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:31.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:31.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:31.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:31.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:31.713 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:35:32.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:35:32.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:35:32.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:32.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi146 2023-07-20T02:35:32.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi146 to 17168M 2023-07-20T02:35:32.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696] boot 2023-07-20T02:35:32.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T02:35:32.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:32 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:32.468 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:35:32.469 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:35:32.469 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:32.469 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi146 2023-07-20T02:35:32.469 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi146 to 17168M 2023-07-20T02:35:32.469 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696] boot 2023-07-20T02:35:32.469 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T02:35:32.470 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:32 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:32.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:35:32.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:35:32.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:35:32.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi146 2023-07-20T02:35:32.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi146 to 17168M 2023-07-20T02:35:32.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: osd.1 [v2:172.21.15.146:6800/3123864696,v1:172.21.15.146:6801/3123864696] boot 2023-07-20T02:35:32.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T02:35:32.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:32 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:35:32.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/3453448898 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b780fe8a0 msgr2=0x7f3b780fed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:32.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/3453448898 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b780fe8a0 0x7f3b780fed00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009990 tx=0x7f3b64009ca0 comp rx=0 tx=0).stop 2023-07-20T02:35:32.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/3453448898 shutdown_connections 2023-07-20T02:35:32.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/3453448898 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b780ff240 0x7f3b78108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:32.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/3453448898 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78101e20 0x7f3b78102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/3453448898 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b780fe8a0 0x7f3b780fed00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/3453448898 >> 172.21.15.151:0/3453448898 conn(0x7f3b780fa420 msgr2=0x7f3b780fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:35:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/3453448898 shutdown_connections 2023-07-20T02:35:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/3453448898 wait complete. 2023-07-20T02:35:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 Processor -- start 2023-07-20T02:35:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.910+0000 7f3b7f0fa700 1 -- start start 2023-07-20T02:35:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7f0fa700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b780fe8a0 0x7f3b78101520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7f0fa700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b780ff240 0x7f3b780ffb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7f0fa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b78101e20 0x7f3b781000b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7f0fa700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b78100830 con 0x7f3b780fe8a0 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7f0fa700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b781009a0 con 0x7f3b780ff240 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7f0fa700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b781a0490 con 0x7f3b78101e20 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b78101e20 0x7f3b781000b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b78101e20 0x7f3b781000b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40752/0 (socket says 172.21.15.151:40752) 2023-07-20T02:35:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 -- 172.21.15.151:0/408255058 learned_addr learned my addr 172.21.15.151:0/408255058 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7ce96700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b780fe8a0 0x7f3b78101520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b77fff700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b780ff240 0x7f3b780ffb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 -- 172.21.15.151:0/408255058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b780ff240 msgr2=0x7f3b780ffb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b780ff240 0x7f3b780ffb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 -- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b780fe8a0 msgr2=0x7f3b78101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b780fe8a0 0x7f3b78101520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 -- 172.21.15.151:0/408255058 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b640095f0 con 0x7f3b78101e20 2023-07-20T02:35:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7d697700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b78101e20 0x7f3b781000b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3b70007d70 tx=0x7f3b7000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:35:32.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b75ffb700 1 -- 172.21.15.151:0/408255058 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000ad40 con 0x7f3b78101e20 2023-07-20T02:35:32.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b75ffb700 1 -- 172.21.15.151:0/408255058 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b700128e0 con 0x7f3b78101e20 2023-07-20T02:35:32.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b781a0770 con 0x7f3b78101e20 2023-07-20T02:35:32.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.911+0000 7f3b75ffb700 1 -- 172.21.15.151:0/408255058 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b700116b0 con 0x7f3b78101e20 2023-07-20T02:35:32.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.912+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b781a0c70 con 0x7f3b78101e20 2023-07-20T02:35:32.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.913+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b78107fe0 con 0x7f3b78101e20 2023-07-20T02:35:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.914+0000 7f3b75ffb700 1 -- 172.21.15.151:0/408255058 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f3b70012420 con 0x7f3b78101e20 2023-07-20T02:35:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.914+0000 7f3b75ffb700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b680a10c0 0x7f3b680a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.914+0000 7f3b75ffb700 1 -- 172.21.15.151:0/408255058 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f3b700cb2f0 con 0x7f3b78101e20 2023-07-20T02:35:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.914+0000 7f3b7ce96700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b680a10c0 0x7f3b680a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.915+0000 7f3b7ce96700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b680a10c0 0x7f3b680a3580 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b78067f10 tx=0x7f3b6c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:35:32.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:32.919+0000 7f3b75ffb700 1 -- 172.21.15.151:0/408255058 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3b70096290 con 0x7f3b78101e20 2023-07-20T02:35:33.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:33.110+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b78060bb0 con 0x7f3b680a10c0 2023-07-20T02:35:33.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:33 smithi017 ceph-mon[127345]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T02:35:33.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:33.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:33.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:33.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:33 smithi146 ceph-mon[128841]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T02:35:33.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:33.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:33.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:33.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:33 smithi151 ceph-mon[129163]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T02:35:33.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:33.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:33.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:34.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:34 smithi017 ceph-mon[127345]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:34.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:34 smithi017 ceph-mon[127345]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:34.467 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:34 smithi151 ceph-mon[129163]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:34.468 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:34 smithi151 ceph-mon[129163]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:34.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:34 smithi146 ceph-mon[128841]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:34.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:34 smithi146 ceph-mon[128841]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:36 smithi151 ceph-mon[129163]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:36 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/1218029294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]: dispatch 2023-07-20T02:35:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:36 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]: dispatch 2023-07-20T02:35:36.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:36 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]': finished 2023-07-20T02:35:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:36 smithi151 ceph-mon[129163]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T02:35:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:36.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:36 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/2337046058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:36.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:36 smithi017 ceph-mon[127345]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:36.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:36 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/1218029294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]: dispatch 2023-07-20T02:35:36.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:36 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]: dispatch 2023-07-20T02:35:36.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:36 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]': finished 2023-07-20T02:35:36.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:36 smithi017 ceph-mon[127345]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T02:35:36.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:36.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:36 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/2337046058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:36.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:36 smithi146 ceph-mon[128841]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:36.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:36 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/1218029294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]: dispatch 2023-07-20T02:35:36.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:36 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]: dispatch 2023-07-20T02:35:36.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:36 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a08d37e7-a3cd-4694-9441-b505bd167c47"}]': finished 2023-07-20T02:35:36.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:36 smithi146 ceph-mon[128841]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T02:35:36.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:36.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:36 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/2337046058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:38.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:38 smithi151 ceph-mon[129163]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:38.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:38 smithi017 ceph-mon[127345]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:38.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:38 smithi146 ceph-mon[128841]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:40.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:40 smithi151 ceph-mon[129163]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:40.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:40 smithi017 ceph-mon[127345]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:40 smithi146 ceph-mon[128841]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:42.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:42 smithi151 ceph-mon[129163]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:42.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:42 smithi017 ceph-mon[127345]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:42.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:42 smithi146 ceph-mon[128841]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:43.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:43 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T02:35:43.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:43 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:43.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:43 smithi151 ceph-mon[129163]: Deploying daemon osd.2 on smithi151 2023-07-20T02:35:43.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:43 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T02:35:43.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:43 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:43.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:43 smithi017 ceph-mon[127345]: Deploying daemon osd.2 on smithi151 2023-07-20T02:35:43.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:43 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T02:35:43.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:43 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:43.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:43 smithi146 ceph-mon[128841]: Deploying daemon osd.2 on smithi151 2023-07-20T02:35:44.563 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:44 smithi151 ceph-mon[129163]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:44.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:44 smithi017 ceph-mon[127345]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:44.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:44 smithi146 ceph-mon[128841]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:46.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:46 smithi151 ceph-mon[129163]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:46.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:46 smithi017 ceph-mon[127345]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:46.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:46 smithi146 ceph-mon[128841]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:48.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:48.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:48.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:47 smithi151 ceph-mon[129163]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:48.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:48.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:48.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:48.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:47 smithi017 ceph-mon[127345]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:48.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:48.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:48.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:48.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:47 smithi146 ceph-mon[128841]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:48.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:49.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.827+0000 7f3b75ffb700 1 -- 172.21.15.151:0/408255058 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3b78060bb0 con 0x7f3b680a10c0 2023-07-20T02:35:49.823 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 2 on host 'smithi151' 2023-07-20T02:35:49.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b680a10c0 msgr2=0x7f3b680a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:49.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b680a10c0 0x7f3b680a3580 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b78067f10 tx=0x7f3b6c007400 comp rx=0 tx=0).stop 2023-07-20T02:35:49.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b78101e20 msgr2=0x7f3b781000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b78101e20 0x7f3b781000b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3b70007d70 tx=0x7f3b7000a340 comp rx=0 tx=0).stop 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 shutdown_connections 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f3b680a10c0 0x7f3b680a3580 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b78101e20 0x7f3b781000b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f3b780fe8a0 0x7f3b78101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 --2- 172.21.15.151:0/408255058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b780ff240 0x7f3b780ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 >> 172.21.15.151:0/408255058 conn(0x7f3b780fa420 msgr2=0x7f3b78105120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:35:49.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 shutdown_connections 2023-07-20T02:35:49.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:35:49.830+0000 7f3b7f0fa700 1 -- 172.21.15.151:0/408255058 wait complete. 2023-07-20T02:35:50.005 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.006 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.006 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:50.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:50.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:50.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:50.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.297 DEBUG:teuthology.orchestra.run.smithi151:osd.2> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.2.service 2023-07-20T02:35:50.299 INFO:tasks.cephadm:Deploying osd.3 on smithi017 with /dev/vg_nvme/lv_3... 2023-07-20T02:35:50.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T02:35:50.331 INFO:journalctl@ceph.osd.2.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:35:50.876 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:35:50.973 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-2[134964]: 2023-07-20T02:35:50.800+0000 7f12b8263640 -1 osd.2 0 log_to_monitors true 2023-07-20T02:35:50.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:50 smithi151 ceph-mon[129163]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:50.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:50.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:51.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:50 smithi146 ceph-mon[128841]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:51.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:51.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:51.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:51.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:50 smithi017 ceph-mon[127345]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:51.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:51.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:51.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:35:51.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:51 smithi017 ceph-mon[127345]: from='osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:35:51.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:51 smithi017 ceph-mon[127345]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:35:52.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:51 smithi146 ceph-mon[128841]: from='osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:35:52.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:51 smithi146 ceph-mon[128841]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:35:52.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:51 smithi151 ceph-mon[129163]: from='osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:35:52.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:51 smithi151 ceph-mon[129163]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:35:52.325 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T02:35:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T02:35:52.326 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2023-07-20T02:35:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2023-07-20T02:35:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00813653 s, 1.3 GB/s 2023-07-20T02:35:52.326 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2023-07-20T02:35:52.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi017:vg_nvme/lv_3 2023-07-20T02:35:53.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:53.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T02:35:53.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T02:35:53.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:53.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:35:53.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:35:53.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:53.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:53.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:53.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:53.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T02:35:53.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T02:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:53.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:53.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.109 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-2[134964]: 2023-07-20T02:35:52.808+0000 7f12ad0da700 -1 osd.2 0 waiting for initial osdmap 2023-07-20T02:35:53.109 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-2[134964]: 2023-07-20T02:35:52.813+0000 7f12a6ecc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:35:53.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:53.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T02:35:53.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T02:35:53.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:53.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:35:53.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:35:53.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:35:53.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:53.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:35:53.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:53.389 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:35:53.864 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:53 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi151 2023-07-20T02:35:53.865 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:53 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi151 to 17168M 2023-07-20T02:35:53.865 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:53 smithi017 ceph-mon[127345]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:35:53.865 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:53 smithi017 ceph-mon[127345]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T02:35:53.865 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:53.865 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:54.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:53 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi151 2023-07-20T02:35:54.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:53 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi151 to 17168M 2023-07-20T02:35:54.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:53 smithi146 ceph-mon[128841]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:35:54.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:53 smithi146 ceph-mon[128841]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T02:35:54.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:54.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:54.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:53 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi151 2023-07-20T02:35:54.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:53 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi151 to 17168M 2023-07-20T02:35:54.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:53 smithi151 ceph-mon[129163]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:35:54.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:53 smithi151 ceph-mon[129163]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T02:35:54.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:54.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:54.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 -- 172.21.15.17:0/2365346751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 msgr2=0x7fe538100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:54.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 --2- 172.21.15.17:0/2365346751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538100d60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0099c0 tx=0x7fe52c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:35:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 -- 172.21.15.17:0/2365346751 shutdown_connections 2023-07-20T02:35:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 --2- 172.21.15.17:0/2365346751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5381012a0 0x7fe538107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 --2- 172.21.15.17:0/2365346751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5380ff700 0x7fe5380ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 --2- 172.21.15.17:0/2365346751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538100d60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 -- 172.21.15.17:0/2365346751 >> 172.21.15.17:0/2365346751 conn(0x7fe5380fac80 msgr2=0x7fe5380fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:35:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 -- 172.21.15.17:0/2365346751 shutdown_connections 2023-07-20T02:35:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.452+0000 7fe53fbac700 1 -- 172.21.15.17:0/2365346751 wait complete. 2023-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 Processor -- start 2023-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 -- start start 2023-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5380ff700 0x7fe538195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5381012a0 0x7fe53819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53819cfd0 con 0x7fe538100900 2023-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53819d110 con 0x7fe5381012a0 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53fbac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53819d2a0 con 0x7fe5380ff700 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53d147700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53d147700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42778/0 (socket says 172.21.15.17:42778) 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53d147700 1 -- 172.21.15.17:0/3127492219 learned_addr learned my addr 172.21.15.17:0/3127492219 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53d948700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5380ff700 0x7fe538195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.453+0000 7fe53e149700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5381012a0 0x7fe53819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53d147700 1 -- 172.21.15.17:0/3127492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5380ff700 msgr2=0x7fe538195440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53d147700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5380ff700 0x7fe538195440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:54.455 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53d147700 1 -- 172.21.15.17:0/3127492219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5381012a0 msgr2=0x7fe53819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:35:54.455 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53d147700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5381012a0 0x7fe53819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:35:54.455 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53d147700 1 -- 172.21.15.17:0/3127492219 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c0095f0 con 0x7fe538100900 2023-07-20T02:35:54.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53d147700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538195980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe52c003af0 tx=0x7fe52c003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:35:54.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c01d070 con 0x7fe538100900 2023-07-20T02:35:54.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe52c004640 con 0x7fe538100900 2023-07-20T02:35:54.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c00f460 con 0x7fe538100900 2023-07-20T02:35:54.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53819d520 con 0x7fe538100900 2023-07-20T02:35:54.456 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.454+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53819d970 con 0x7fe538100900 2023-07-20T02:35:54.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.455+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe538106cf0 con 0x7fe538100900 2023-07-20T02:35:54.457 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.456+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fe52c003f50 con 0x7fe538100900 2023-07-20T02:35:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.457+0000 7fe52affd700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe5240a10c0 0x7fe5240a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:35:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.457+0000 7fe53d948700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe5240a10c0 0x7fe5240a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:35:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.457+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2851+0+0 (secure 0 0 0) 0x7fe52c0cc5e0 con 0x7fe538100900 2023-07-20T02:35:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.457+0000 7fe53d948700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe5240a10c0 0x7fe5240a3580 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe538100760 tx=0x7fe53400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:35:54.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.461+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe52c05c780 con 0x7fe538100900 2023-07-20T02:35:54.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:54.666+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe538060bb0 con 0x7fe5240a10c0 2023-07-20T02:35:55.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:35:55.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:35:55.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:55.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358] boot 2023-07-20T02:35:55.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T02:35:55.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:55.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:55.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:55.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:55.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:55.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358] boot 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:55.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:55.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:55.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:35:55.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:35:55.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:35:55.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: osd.2 [v2:172.21.15.151:6800/1299864358,v1:172.21.15.151:6801/1299864358] boot 2023-07-20T02:35:55.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T02:35:55.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:35:55.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:35:55.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:35:55.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:35:55.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:35:56.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:55 smithi151 ceph-mon[129163]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T02:35:56.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:55 smithi151 ceph-mon[129163]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:35:56.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:55 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:35:56.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:55 smithi017 ceph-mon[127345]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T02:35:56.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:55 smithi017 ceph-mon[127345]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:35:56.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:55 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:35:56.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:55 smithi146 ceph-mon[128841]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T02:35:56.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:55 smithi146 ceph-mon[128841]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:35:56.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:55 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:35:57.106 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:56 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T02:35:57.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:56 smithi151 ceph-mon[129163]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T02:35:57.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:56 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:35:57.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:56 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T02:35:57.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:56 smithi017 ceph-mon[127345]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T02:35:57.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:56 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:35:57.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:56 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T02:35:57.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:56 smithi146 ceph-mon[128841]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T02:35:57.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:56 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:35:57.698 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143459]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:35:57.699 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143459]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:35:57.699 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143459]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:35:57.699 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143459]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:35:57.955 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:35:57.959+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe52c0977c0 con 0x7fe538100900 2023-07-20T02:35:58.095 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T02:35:58.095 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T02:35:58.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:35:58.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1413021786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62ad61a1-05b0-4e38-a2eb-530485d806b2"}]: dispatch 2023-07-20T02:35:58.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1413021786' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62ad61a1-05b0-4e38-a2eb-530485d806b2"}]': finished 2023-07-20T02:35:58.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T02:35:58.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:35:58.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:57 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2476294056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:58.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T02:35:58.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T02:35:58.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:35:58.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1413021786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62ad61a1-05b0-4e38-a2eb-530485d806b2"}]: dispatch 2023-07-20T02:35:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1413021786' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62ad61a1-05b0-4e38-a2eb-530485d806b2"}]': finished 2023-07-20T02:35:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T02:35:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:35:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:57 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2476294056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:58.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T02:35:58.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T02:35:58.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:35:58.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1413021786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62ad61a1-05b0-4e38-a2eb-530485d806b2"}]: dispatch 2023-07-20T02:35:58.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1413021786' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62ad61a1-05b0-4e38-a2eb-530485d806b2"}]': finished 2023-07-20T02:35:58.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T02:35:58.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:35:58.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:57 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2476294056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:35:58.200 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143513]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:35:58.201 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143513]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:35:58.201 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143513]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:35:58.201 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:35:57 smithi017 sudo[143513]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:35:58.347 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136116]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:35:58.348 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:35:58.348 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136116]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:35:58.348 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136116]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:35:58.812 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136119]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:35:58.813 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:35:58.813 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136119]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:35:58.813 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:35:58 smithi146 sudo[136119]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:35:58.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:58.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:58 smithi151 ceph-mon[129163]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T02:35:58.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:35:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:58.869 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137243]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:35:58.869 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137243]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:35:58.869 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137243]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:35:58.869 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137243]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:35:59.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:59.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:58 smithi017 ceph-mon[127345]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T02:35:59.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:59.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:59.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:58 smithi146 ceph-mon[128841]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T02:35:59.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:35:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:35:59.356 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137246]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:35:59.357 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137246]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:35:59.357 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137246]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:35:59.357 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:35:58 smithi151 sudo[137246]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:35:59.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144267]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:35:59.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144267]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:35:59.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144267]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:35:59.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144267]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:36:00.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144466]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:36:00.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144466]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:36:00.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144466]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:36:00.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:35:59 smithi017 sudo[144466]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:36:00.293 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137249]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:36:00.294 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137249]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:36:00.294 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137249]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:36:00.294 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137249]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:36:00.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137252]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:36:00.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137252]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:36:00.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137252]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:36:00.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 sudo[137252]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:00.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:00.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:00.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:00.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:00.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:00.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136122]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:36:00.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136122]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:36:00.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136122]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:36:00.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136122]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:36:01.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136125]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:36:01.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136125]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:36:01.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136125]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:36:01.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 sudo[136125]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:01.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:01.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:01.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:01.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:01 smithi146 ceph-mon[128841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:01.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:01 smithi146 ceph-mon[128841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:01.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:01 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:01.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:01 smithi146 ceph-mon[128841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:01.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:01 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:01.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:01 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:01.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:01 smithi146 ceph-mon[128841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:01.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:01 smithi151 ceph-mon[129163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:01.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:01 smithi151 ceph-mon[129163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:01.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:01 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:01.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:01 smithi151 ceph-mon[129163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:01.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:01 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:01.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:01 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:01.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:01 smithi151 ceph-mon[129163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:01.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:01 smithi017 ceph-mon[127345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:01.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:01 smithi017 ceph-mon[127345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:01.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:01 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:36:01.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:01 smithi017 ceph-mon[127345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:36:01.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:01 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:36:01.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:01 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:36:01.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:01 smithi017 ceph-mon[127345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:36:02.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:02 smithi146 ceph-mon[128841]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:02.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:02 smithi151 ceph-mon[129163]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:02.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:02 smithi017 ceph-mon[127345]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:04.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:04 smithi146 ceph-mon[128841]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-20T02:36:04.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:04 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T02:36:04.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:04 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:04.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:04 smithi146 ceph-mon[128841]: Deploying daemon osd.3 on smithi017 2023-07-20T02:36:04.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:04 smithi151 ceph-mon[129163]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-20T02:36:04.607 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:04 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T02:36:04.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:04 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:04.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:04 smithi151 ceph-mon[129163]: Deploying daemon osd.3 on smithi017 2023-07-20T02:36:04.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:04 smithi017 ceph-mon[127345]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-20T02:36:04.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:04 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T02:36:04.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:04 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:04.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:04 smithi017 ceph-mon[127345]: Deploying daemon osd.3 on smithi017 2023-07-20T02:36:05.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:05 smithi017 ceph-mon[127345]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-20T02:36:06.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:05 smithi146 ceph-mon[128841]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-20T02:36:06.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:05 smithi151 ceph-mon[129163]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-20T02:36:07.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:07 smithi017 ceph-mon[127345]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-20T02:36:08.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:07 smithi146 ceph-mon[128841]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-20T02:36:08.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:07 smithi151 ceph-mon[129163]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-20T02:36:09.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:09.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:09.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:09.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:09.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:09.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:09.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:09.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:09.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:10.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:10 smithi146 ceph-mon[128841]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-20T02:36:10.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:10 smithi017 ceph-mon[127345]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-20T02:36:10.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:10 smithi151 ceph-mon[129163]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-20T02:36:10.665 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 3 on host 'smithi017' 2023-07-20T02:36:10.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.669+0000 7fe52affd700 1 -- 172.21.15.17:0/3127492219 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe538060bb0 con 0x7fe5240a10c0 2023-07-20T02:36:10.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe5240a10c0 msgr2=0x7fe5240a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:10.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe5240a10c0 0x7fe5240a3580 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe538100760 tx=0x7fe53400a380 comp rx=0 tx=0).stop 2023-07-20T02:36:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 msgr2=0x7fe538195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538195980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe52c003af0 tx=0x7fe52c003b20 comp rx=0 tx=0).stop 2023-07-20T02:36:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 shutdown_connections 2023-07-20T02:36:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe5240a10c0 0x7fe5240a3580 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5381012a0 0x7fe53819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5380ff700 0x7fe538195440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:10.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 --2- 172.21.15.17:0/3127492219 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe538100900 0x7fe538195980 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 >> 172.21.15.17:0/3127492219 conn(0x7fe5380fac80 msgr2=0x7fe538103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:36:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 shutdown_connections 2023-07-20T02:36:10.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:10.671+0000 7fe53fbac700 1 -- 172.21.15.17:0/3127492219 wait complete. 2023-07-20T02:36:11.081 DEBUG:teuthology.orchestra.run.smithi017:osd.3> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.3.service 2023-07-20T02:36:11.083 INFO:tasks.cephadm:Deploying osd.4 on smithi146 with /dev/vg_nvme/lv_3... 2023-07-20T02:36:11.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T02:36:11.113 INFO:journalctl@ceph.osd.3.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:36:11.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:11.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:11.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:11 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:11.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:11.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:11.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:11.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:11.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:11.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:11.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:11 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:36:12.199 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:36:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:36:11.974+0000 7f050edfa640 -1 osd.3 0 log_to_monitors true 2023-07-20T02:36:12.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:12 smithi017 ceph-mon[127345]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-20T02:36:12.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:12 smithi017 ceph-mon[127345]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T02:36:13.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:12 smithi146 ceph-mon[128841]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-20T02:36:13.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:12 smithi146 ceph-mon[128841]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T02:36:13.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:12 smithi151 ceph-mon[129163]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-20T02:36:13.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:12 smithi151 ceph-mon[129163]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T02:36:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T02:36:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T02:36:13.273 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2023-07-20T02:36:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2023-07-20T02:36:13.273 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110668 s, 947 MB/s 2023-07-20T02:36:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2023-07-20T02:36:13.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2023-07-20T02:36:13.949 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:36:13.651+0000 7f0503c71700 -1 osd.3 0 waiting for initial osdmap 2023-07-20T02:36:13.949 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:36:13.658+0000 7f04fe264700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:36:13.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T02:36:13.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T02:36:13.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:13.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:36:13.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:13.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:13.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:13.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:13.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:13.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:13.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:13.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:13 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T02:36:14.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T02:36:14.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:14.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:36:14.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:14.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:14.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:14.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:14.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:13 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T02:36:14.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T02:36:14.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:14.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:14.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:13 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:14.259 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:36:14.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:14 smithi146 ceph-mon[128841]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:14.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:14 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi017 2023-07-20T02:36:14.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:14 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi017 to 8586M 2023-07-20T02:36:14.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:14 smithi146 ceph-mon[128841]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:36:14.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:14 smithi146 ceph-mon[128841]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T02:36:14.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:14 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:14.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:14 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:14.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:14 smithi017 ceph-mon[127345]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:14.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:14 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi017 2023-07-20T02:36:14.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:14 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi017 to 8586M 2023-07-20T02:36:14.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:14 smithi017 ceph-mon[127345]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:36:14.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:14 smithi017 ceph-mon[127345]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T02:36:14.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:14 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:14.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:14 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:15.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:14 smithi151 ceph-mon[129163]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:15.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:14 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi017 2023-07-20T02:36:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:14 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi017 to 8586M 2023-07-20T02:36:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:14 smithi151 ceph-mon[129163]: from='osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:36:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:14 smithi151 ceph-mon[129163]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T02:36:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:14 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:14 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:15.601 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 -- 172.21.15.146:0/336326157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4440fffe0 msgr2=0x7fe444100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 --2- 172.21.15.146:0/336326157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4440fffe0 0x7fe444100460 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe438009990 tx=0x7fe438009ca0 comp rx=0 tx=0).stop 2023-07-20T02:36:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 -- 172.21.15.146:0/336326157 shutdown_connections 2023-07-20T02:36:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 --2- 172.21.15.146:0/336326157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4441009a0 0x7fe4441072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 --2- 172.21.15.146:0/336326157 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4440fee80 0x7fe4440ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 --2- 172.21.15.146:0/336326157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4440fffe0 0x7fe444100460 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 -- 172.21.15.146:0/336326157 >> 172.21.15.146:0/336326157 conn(0x7fe4440fa420 msgr2=0x7fe4440fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:36:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 -- 172.21.15.146:0/336326157 shutdown_connections 2023-07-20T02:36:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.606+0000 7fe44888a700 1 -- 172.21.15.146:0/336326157 wait complete. 2023-07-20T02:36:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 Processor -- start 2023-07-20T02:36:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 -- start start 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4440fee80 0x7fe444194be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4440fffe0 0x7fe444195120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4441009a0 0x7fe44419c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44419c770 con 0x7fe4440fee80 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44419c8b0 con 0x7fe4441009a0 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44888a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44419ca40 con 0x7fe4440fffe0 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe442d9d700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4441009a0 0x7fe44419c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe442d9d700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4441009a0 0x7fe44419c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:52220/0 (socket says 172.21.15.146:52220) 2023-07-20T02:36:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe442d9d700 1 -- 172.21.15.146:0/1209380565 learned_addr learned my addr 172.21.15.146:0/1209380565 (peer_addr_for_me v2:172.21.15.146:0/0) 2023-07-20T02:36:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe441d9b700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4440fffe0 0x7fe444195120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe44259c700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4440fee80 0x7fe444194be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.607+0000 7fe442d9d700 1 -- 172.21.15.146:0/1209380565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4440fffe0 msgr2=0x7fe444195120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe442d9d700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4440fffe0 0x7fe444195120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe442d9d700 1 -- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4440fee80 msgr2=0x7fe444194be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe442d9d700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4440fee80 0x7fe444194be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe442d9d700 1 -- 172.21.15.146:0/1209380565 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4380095f0 con 0x7fe4441009a0 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe44259c700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4440fee80 0x7fe444194be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe442d9d700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4441009a0 0x7fe44419c1a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00ada0 tx=0x7fe43c00c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe4337fe700 1 -- 172.21.15.146:0/1209380565 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00f070 con 0x7fe4441009a0 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44419cd20 con 0x7fe4441009a0 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe4337fe700 1 -- 172.21.15.146:0/1209380565 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe43c007b30 con 0x7fe4441009a0 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe4337fe700 1 -- 172.21.15.146:0/1209380565 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c0054e0 con 0x7fe4441009a0 2023-07-20T02:36:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.608+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44419d240 con 0x7fe4441009a0 2023-07-20T02:36:15.608 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.609+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4441064b0 con 0x7fe4441009a0 2023-07-20T02:36:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.610+0000 7fe4337fe700 1 -- 172.21.15.146:0/1209380565 <== mon.2 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe43c004ad0 con 0x7fe4441009a0 2023-07-20T02:36:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.611+0000 7fe4337fe700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe42c0a2260 0x7fe42c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.611+0000 7fe4337fe700 1 -- 172.21.15.146:0/1209380565 <== mon.2 v2:172.21.15.146:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fe43c0cad40 con 0x7fe4441009a0 2023-07-20T02:36:15.610 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.611+0000 7fe44259c700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe42c0a2260 0x7fe42c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:15.610 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.611+0000 7fe44259c700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe42c0a2260 0x7fe42c0a4720 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe434007900 tx=0x7fe434008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:36:15.610 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.615+0000 7fe4337fe700 1 -- 172.21.15.146:0/1209380565 <== mon.2 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe43c095720 con 0x7fe4441009a0 2023-07-20T02:36:15.805 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:15.809+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe444060bb0 con 0x7fe42c0a2260 2023-07-20T02:36:15.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:15 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:36:15.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:15 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:36:15.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:15 smithi017 ceph-mon[127345]: osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850] boot 2023-07-20T02:36:15.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:15 smithi017 ceph-mon[127345]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T02:36:15.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:15 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:15.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:15 smithi017 ceph-mon[127345]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T02:36:16.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:15 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:36:16.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:15 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:36:16.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:15 smithi146 ceph-mon[128841]: osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850] boot 2023-07-20T02:36:16.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:15 smithi146 ceph-mon[128841]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T02:36:16.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:15 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:16.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:15 smithi146 ceph-mon[128841]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T02:36:16.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:15 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:36:16.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:15 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:36:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:15 smithi151 ceph-mon[129163]: osd.3 [v2:172.21.15.17:6810/604342850,v1:172.21.15.17:6811/604342850] boot 2023-07-20T02:36:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:15 smithi151 ceph-mon[129163]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T02:36:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:15 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:36:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:15 smithi151 ceph-mon[129163]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T02:36:16.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:16 smithi146 ceph-mon[128841]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:16.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:16 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:36:16.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:16 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:36:16.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:16 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:16.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:16 smithi146 ceph-mon[128841]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T02:36:16.998 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:16 smithi017 ceph-mon[127345]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:17.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:16 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:36:17.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:16 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:36:17.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:16 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:17.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:16 smithi017 ceph-mon[127345]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T02:36:17.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:16 smithi151 ceph-mon[129163]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:36:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:16 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:36:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:16 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:36:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:16 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:16 smithi151 ceph-mon[129163]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T02:36:17.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:17 smithi146 ceph-mon[128841]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:36:17.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:17 smithi146 ceph-mon[128841]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T02:36:17.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:17 smithi017 ceph-mon[127345]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:36:17.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:17 smithi017 ceph-mon[127345]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T02:36:18.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:17 smithi151 ceph-mon[129163]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:36:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:17 smithi151 ceph-mon[129163]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T02:36:18.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:18 smithi017 ceph-mon[127345]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:18.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:18 smithi017 ceph-mon[127345]: osdmap e29: 4 total, 4 up, 4 in 2023-07-20T02:36:18.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:18 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/1019227418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]: dispatch 2023-07-20T02:36:18.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:18 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]: dispatch 2023-07-20T02:36:18.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:18 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]': finished 2023-07-20T02:36:18.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:18 smithi017 ceph-mon[127345]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T02:36:18.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:18 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:19.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:18 smithi146 ceph-mon[128841]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:19.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:18 smithi146 ceph-mon[128841]: osdmap e29: 4 total, 4 up, 4 in 2023-07-20T02:36:19.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:18 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/1019227418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]: dispatch 2023-07-20T02:36:19.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:18 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]: dispatch 2023-07-20T02:36:19.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:18 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]': finished 2023-07-20T02:36:19.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:18 smithi146 ceph-mon[128841]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T02:36:19.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:18 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:19.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:18 smithi151 ceph-mon[129163]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:19.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:18 smithi151 ceph-mon[129163]: osdmap e29: 4 total, 4 up, 4 in 2023-07-20T02:36:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:18 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/1019227418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]: dispatch 2023-07-20T02:36:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:18 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]: dispatch 2023-07-20T02:36:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:18 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ed288f-9c0d-415c-b52f-075d3b7599db"}]': finished 2023-07-20T02:36:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:18 smithi151 ceph-mon[129163]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T02:36:19.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:18 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:19.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:19 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/501544492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:36:20.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:19 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/501544492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:36:20.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:19 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/501544492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:36:20.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:20 smithi017 ceph-mon[127345]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:36:21.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:20 smithi146 ceph-mon[128841]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:36:21.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:20 smithi151 ceph-mon[129163]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:36:22.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:22 smithi017 ceph-mon[127345]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-20T02:36:23.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:22 smithi146 ceph-mon[128841]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-20T02:36:23.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:22 smithi151 ceph-mon[129163]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-20T02:36:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:24 smithi017 ceph-mon[127345]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:36:25.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:24 smithi146 ceph-mon[128841]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:36:25.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:24 smithi151 ceph-mon[129163]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:36:26.059 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:25 smithi146 ceph-mon[128841]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:36:26.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T02:36:26.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:26.060 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:25 smithi146 ceph-mon[128841]: Deploying daemon osd.4 on smithi146 2023-07-20T02:36:26.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:25 smithi151 ceph-mon[129163]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:36:26.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T02:36:26.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:25 smithi151 ceph-mon[129163]: Deploying daemon osd.4 on smithi146 2023-07-20T02:36:26.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:25 smithi017 ceph-mon[127345]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:36:26.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T02:36:26.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:26.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:25 smithi017 ceph-mon[127345]: Deploying daemon osd.4 on smithi146 2023-07-20T02:36:28.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:27 smithi146 ceph-mon[128841]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:28.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:27 smithi017 ceph-mon[127345]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:28.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:27 smithi151 ceph-mon[129163]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:30.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:29 smithi017 ceph-mon[127345]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:30.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:29 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:30.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:29 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:30.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:29 smithi146 ceph-mon[128841]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:30.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:29 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:30.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:29 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:29 smithi151 ceph-mon[129163]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:29 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:29 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:31.059 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:31.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:31.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:32.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:31 smithi017 ceph-mon[127345]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:32.313 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:31 smithi146 ceph-mon[128841]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:32.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:31 smithi151 ceph-mon[129163]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:32.875 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.879+0000 7fe4337fe700 1 -- 172.21.15.146:0/1209380565 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe444060bb0 con 0x7fe42c0a2260 2023-07-20T02:36:32.876 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 4 on host 'smithi146' 2023-07-20T02:36:32.877 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe42c0a2260 msgr2=0x7fe42c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:32.877 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe42c0a2260 0x7fe42c0a4720 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe434007900 tx=0x7fe434008040 comp rx=0 tx=0).stop 2023-07-20T02:36:32.878 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4441009a0 msgr2=0x7fe44419c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:32.878 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4441009a0 0x7fe44419c1a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00ada0 tx=0x7fe43c00c330 comp rx=0 tx=0).stop 2023-07-20T02:36:32.878 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 shutdown_connections 2023-07-20T02:36:32.879 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fe42c0a2260 0x7fe42c0a4720 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:32.879 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4441009a0 0x7fe44419c1a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:32.879 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fe4440fee80 0x7fe444194be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:32.880 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 --2- 172.21.15.146:0/1209380565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4440fffe0 0x7fe444195120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:32.880 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 >> 172.21.15.146:0/1209380565 conn(0x7fe4440fa420 msgr2=0x7fe4441035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:36:32.880 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 shutdown_connections 2023-07-20T02:36:32.881 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:36:32.881+0000 7fe44888a700 1 -- 172.21.15.146:0/1209380565 wait complete. 2023-07-20T02:36:33.361 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.4.service 2023-07-20T02:36:33.364 INFO:tasks.cephadm:Deploying osd.5 on smithi151 with /dev/vg_nvme/lv_3... 2023-07-20T02:36:33.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T02:36:33.395 INFO:journalctl@ceph.osd.4.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:36:33.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:33.930 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:36:34.094 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:36:33 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-4[139220]: 2023-07-20T02:36:33.840+0000 7f7845dd5640 -1 osd.4 0 log_to_monitors true 2023-07-20T02:36:34.451 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:34 smithi151 ceph-mon[129163]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:34.452 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:34 smithi151 ceph-mon[129163]: from='osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:36:34.452 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:34 smithi151 ceph-mon[129163]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:36:34.462 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:34 smithi146 ceph-mon[128841]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:34.462 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:34 smithi146 ceph-mon[128841]: from='osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:36:34.463 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:34 smithi146 ceph-mon[128841]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:36:34.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:34 smithi017 ceph-mon[127345]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:34.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:34 smithi017 ceph-mon[127345]: from='osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:36:34.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:34 smithi017 ceph-mon[127345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:36:35.432 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T02:36:35.432 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T02:36:35.432 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2023-07-20T02:36:35.432 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2023-07-20T02:36:35.433 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146361 s, 716 MB/s 2023-07-20T02:36:35.433 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2023-07-20T02:36:35.837 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T02:36:35.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T02:36:35.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:35.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:36:35.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:36:35.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:35.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi146 2023-07-20T02:36:35.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:35.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:35.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:35.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:35.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi146 to 8584M 2023-07-20T02:36:35.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:35.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:35.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:35.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:35.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi151:vg_nvme/lv_3 2023-07-20T02:36:36.064 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-4[139220]: 2023-07-20T02:36:35.843+0000 7f783ac4c700 -1 osd.4 0 waiting for initial osdmap 2023-07-20T02:36:36.064 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-4[139220]: 2023-07-20T02:36:35.849+0000 7f7837243700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:36:36.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T02:36:36.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T02:36:36.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:36.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:36:36.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:36:36.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:36.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi146 2023-07-20T02:36:36.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:36.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:36.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi146 to 8584M 2023-07-20T02:36:36.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:36.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:36.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T02:36:36.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T02:36:36.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:36.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:36:36.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:36:36.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:36.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi146 2023-07-20T02:36:36.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:36.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:36.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi146 to 8584M 2023-07-20T02:36:36.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:36.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:36.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:36.440 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:36:37.059 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:36:37.060 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: osdmap e32: 5 total, 4 up, 5 in 2023-07-20T02:36:37.060 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:37.060 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:37.060 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:37.060 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:37.060 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:37.060 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:37.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:36:37.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: osdmap e32: 5 total, 4 up, 5 in 2023-07-20T02:36:37.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:37.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:37.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:37.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:37.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:37.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:37.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:36:37.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: osdmap e32: 5 total, 4 up, 5 in 2023-07-20T02:36:37.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:37.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:37.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:37.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:37.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:37.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:37.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 -- 172.21.15.151:0/2117773423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c103c60 msgr2=0x7f999c106050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:37.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/2117773423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c103c60 0x7f999c106050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9990009990 tx=0x7f9990009ca0 comp rx=0 tx=0).stop 2023-07-20T02:36:37.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 -- 172.21.15.151:0/2117773423 shutdown_connections 2023-07-20T02:36:37.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/2117773423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c106590 0x7f999c1089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:37.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/2117773423 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f999c101330 0x7f999c103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:37.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/2117773423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c103c60 0x7f999c106050 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:37.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 -- 172.21.15.151:0/2117773423 >> 172.21.15.151:0/2117773423 conn(0x7f999c0fad00 msgr2=0x7f999c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:36:37.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 -- 172.21.15.151:0/2117773423 shutdown_connections 2023-07-20T02:36:37.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 -- 172.21.15.151:0/2117773423 wait complete. 2023-07-20T02:36:37.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.741+0000 7f99a1b8e700 1 Processor -- start 2023-07-20T02:36:37.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f99a1b8e700 1 -- start start 2023-07-20T02:36:37.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f99a1b8e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c101330 0x7f999c196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:37.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f99a1b8e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f999c103c60 0x7f999c196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:37.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f99a1b8e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c106590 0x7f999c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:37.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f99a1b8e700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c19e0d0 con 0x7f999c103c60 2023-07-20T02:36:37.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f99a1b8e700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c19e210 con 0x7f999c106590 2023-07-20T02:36:37.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f99a1b8e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c19e3a0 con 0x7f999c101330 2023-07-20T02:36:37.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c101330 0x7f999c196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:37.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c101330 0x7f999c196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40830/0 (socket says 172.21.15.151:40830) 2023-07-20T02:36:37.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 -- 172.21.15.151:0/141605448 learned_addr learned my addr 172.21.15.151:0/141605448 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:36:37.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999affd700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f999c103c60 0x7f999c196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:37.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 -- 172.21.15.151:0/141605448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c106590 msgr2=0x7f999c19db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:37.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c106590 0x7f999c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:37.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 -- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f999c103c60 msgr2=0x7f999c196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:37.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f999c103c60 0x7f999c196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:37.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 -- 172.21.15.151:0/141605448 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900095f0 con 0x7f999c101330 2023-07-20T02:36:37.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.742+0000 7f999b7fe700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c101330 0x7f999c196540 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9994007ee0 tx=0x7f999400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:36:37.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.743+0000 7f9998ff9700 1 -- 172.21.15.151:0/141605448 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999400de90 con 0x7f999c101330 2023-07-20T02:36:37.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.743+0000 7f9998ff9700 1 -- 172.21.15.151:0/141605448 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f999400f040 con 0x7f999c101330 2023-07-20T02:36:37.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.743+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c19e680 con 0x7f999c101330 2023-07-20T02:36:37.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.743+0000 7f9998ff9700 1 -- 172.21.15.151:0/141605448 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99940147a0 con 0x7f999c101330 2023-07-20T02:36:37.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.743+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c19eb80 con 0x7f999c101330 2023-07-20T02:36:37.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.744+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c0fc880 con 0x7f999c101330 2023-07-20T02:36:37.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.744+0000 7f9998ff9700 1 -- 172.21.15.151:0/141605448 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9994015aa0 con 0x7f999c101330 2023-07-20T02:36:37.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.745+0000 7f9998ff9700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f99840a1150 0x7f99840a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:37.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.745+0000 7f9998ff9700 1 -- 172.21.15.151:0/141605448 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f99940cb350 con 0x7f999c101330 2023-07-20T02:36:37.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.745+0000 7f999affd700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f99840a1150 0x7f99840a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:37.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.746+0000 7f999affd700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f99840a1150 0x7f99840a3610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9990006010 tx=0x7f999000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:36:37.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.749+0000 7f9998ff9700 1 -- 172.21.15.151:0/141605448 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f999404f780 con 0x7f999c101330 2023-07-20T02:36:37.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:37.945+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f999c060bb0 con 0x7f99840a1150 2023-07-20T02:36:38.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:37 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:36:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:37 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:36:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:37 smithi151 ceph-mon[129163]: osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090] boot 2023-07-20T02:36:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:37 smithi151 ceph-mon[129163]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T02:36:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:37 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:37 smithi151 ceph-mon[129163]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:38.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:37 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:36:38.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:37 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:36:38.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:37 smithi017 ceph-mon[127345]: osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090] boot 2023-07-20T02:36:38.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:37 smithi017 ceph-mon[127345]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T02:36:38.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:37 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:38.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:37 smithi017 ceph-mon[127345]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:38.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:37 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:36:38.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:37 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:36:38.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:37 smithi146 ceph-mon[128841]: osd.4 [v2:172.21.15.146:6808/4024595090,v1:172.21.15.146:6809/4024595090] boot 2023-07-20T02:36:38.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:37 smithi146 ceph-mon[128841]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T02:36:38.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:37 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:36:38.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:37 smithi146 ceph-mon[128841]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:36:39.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:38 smithi151 ceph-mon[129163]: osdmap e34: 5 total, 5 up, 5 in 2023-07-20T02:36:39.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:38 smithi151 ceph-mon[129163]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:36:39.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:36:39.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:36:39.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:39.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:38 smithi017 ceph-mon[127345]: osdmap e34: 5 total, 5 up, 5 in 2023-07-20T02:36:39.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:38 smithi017 ceph-mon[127345]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:36:39.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:36:39.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:36:39.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:39.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:38 smithi146 ceph-mon[128841]: osdmap e34: 5 total, 5 up, 5 in 2023-07-20T02:36:39.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:38 smithi146 ceph-mon[128841]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:36:39.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:36:39.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:36:39.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:40.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:39 smithi151 ceph-mon[129163]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:40.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:39 smithi017 ceph-mon[127345]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:40.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:39 smithi146 ceph-mon[128841]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:40 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/715823427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]: dispatch 2023-07-20T02:36:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:40 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]: dispatch 2023-07-20T02:36:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:40 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]': finished 2023-07-20T02:36:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:40 smithi151 ceph-mon[129163]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T02:36:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:41.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:40 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/715823427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]: dispatch 2023-07-20T02:36:41.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:40 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]: dispatch 2023-07-20T02:36:41.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:40 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]': finished 2023-07-20T02:36:41.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:40 smithi017 ceph-mon[127345]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T02:36:41.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:41.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:40 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/715823427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]: dispatch 2023-07-20T02:36:41.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:40 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]: dispatch 2023-07-20T02:36:41.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:40 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7da194cb-67fb-4c73-b84c-fa899c3635b8"}]': finished 2023-07-20T02:36:41.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:40 smithi146 ceph-mon[128841]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T02:36:41.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:42.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:41 smithi017 ceph-mon[127345]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:42.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:41 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/188159780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:36:42.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:41 smithi146 ceph-mon[128841]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:42.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:41 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/188159780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:36:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:41 smithi151 ceph-mon[129163]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:41 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/188159780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:36:44.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:43 smithi017 ceph-mon[127345]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:44.234 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:43 smithi151 ceph-mon[129163]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:44.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:43 smithi146 ceph-mon[128841]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:45.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:45 smithi151 ceph-mon[129163]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:46.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:45 smithi017 ceph-mon[127345]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:46.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:45 smithi146 ceph-mon[128841]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:48.041 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:47 smithi151 ceph-mon[129163]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:48.042 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T02:36:48.042 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:48.042 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:47 smithi151 ceph-mon[129163]: Deploying daemon osd.5 on smithi151 2023-07-20T02:36:48.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:47 smithi017 ceph-mon[127345]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:48.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T02:36:48.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:48.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:47 smithi017 ceph-mon[127345]: Deploying daemon osd.5 on smithi151 2023-07-20T02:36:48.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:47 smithi146 ceph-mon[128841]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:48.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T02:36:48.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:48.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:47 smithi146 ceph-mon[128841]: Deploying daemon osd.5 on smithi151 2023-07-20T02:36:50.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:49 smithi017 ceph-mon[127345]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:50.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:49 smithi146 ceph-mon[128841]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:50.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:49 smithi151 ceph-mon[129163]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:52 smithi151 ceph-mon[129163]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:52.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:52 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:52.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:52 smithi017 ceph-mon[127345]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:52.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:52.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:52.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:52 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:52.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:52 smithi146 ceph-mon[128841]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:52.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:52.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:52.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:52 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:54.205 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.207+0000 7f9998ff9700 1 -- 172.21.15.151:0/141605448 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f999c060bb0 con 0x7f99840a1150 2023-07-20T02:36:54.206 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 5 on host 'smithi151' 2023-07-20T02:36:54.208 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.209+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f99840a1150 msgr2=0x7f99840a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:54.208 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.209+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f99840a1150 0x7f99840a3610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9990006010 tx=0x7f999000b540 comp rx=0 tx=0).stop 2023-07-20T02:36:54.208 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c101330 msgr2=0x7f999c196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:54.208 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c101330 0x7f999c196540 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9994007ee0 tx=0x7f999400d3b0 comp rx=0 tx=0).stop 2023-07-20T02:36:54.208 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 shutdown_connections 2023-07-20T02:36:54.208 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f99840a1150 0x7f99840a3610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:54.208 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c106590 0x7f999c19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:54.209 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f999c101330 0x7f999c196540 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:54.209 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 --2- 172.21.15.151:0/141605448 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f999c103c60 0x7f999c196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:54.209 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 >> 172.21.15.151:0/141605448 conn(0x7f999c0fad00 msgr2=0x7f999c101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:36:54.209 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 shutdown_connections 2023-07-20T02:36:54.209 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:36:54.210+0000 7f99a1b8e700 1 -- 172.21.15.151:0/141605448 wait complete. 2023-07-20T02:36:54.634 DEBUG:teuthology.orchestra.run.smithi151:osd.5> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.5.service 2023-07-20T02:36:54.637 INFO:tasks.cephadm:Deploying osd.6 on smithi017 with /dev/vg_nvme/lv_2... 2023-07-20T02:36:54.637 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T02:36:54.678 INFO:journalctl@ceph.osd.5.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:36:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:54.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:54 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:54.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:54.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:54.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:54.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:54.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:54 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:55.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:55.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:55.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:55.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:55.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:55.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:55.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:55.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:36:55.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:54 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:55.217 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:36:55.498 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:36:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:36:55.396+0000 7f926a383640 -1 osd.5 0 log_to_monitors true 2023-07-20T02:36:55.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:55 smithi017 ceph-mon[127345]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:55.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:55 smithi017 ceph-mon[127345]: from='osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:36:55.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:55 smithi017 ceph-mon[127345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:36:56.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:55 smithi151 ceph-mon[129163]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:56.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:55 smithi151 ceph-mon[129163]: from='osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:36:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:55 smithi151 ceph-mon[129163]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:36:56.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:55 smithi146 ceph-mon[128841]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:56.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:55 smithi146 ceph-mon[128841]: from='osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:36:56.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:55 smithi146 ceph-mon[128841]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:36:56.612 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T02:36:56.613 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T02:36:56.613 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2023-07-20T02:36:56.613 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2023-07-20T02:36:56.613 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00815848 s, 1.3 GB/s 2023-07-20T02:36:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2023-07-20T02:36:56.858 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:36:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:36:56.833+0000 7f925f1fa700 -1 osd.5 0 waiting for initial osdmap 2023-07-20T02:36:56.859 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:36:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:36:56.840+0000 7f9258fec700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:36:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:56 smithi151 ceph-mon[129163]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T02:36:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:56 smithi151 ceph-mon[129163]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T02:36:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:56 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:56 smithi151 ceph-mon[129163]: from='osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:36:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:56 smithi151 ceph-mon[129163]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:36:56.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:56 smithi017 ceph-mon[127345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T02:36:56.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:56 smithi017 ceph-mon[127345]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T02:36:56.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:56 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:56.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:56 smithi017 ceph-mon[127345]: from='osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:36:56.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:56 smithi017 ceph-mon[127345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:36:56.964 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi017:vg_nvme/lv_2 2023-07-20T02:36:57.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:56 smithi146 ceph-mon[128841]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T02:36:57.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:56 smithi146 ceph-mon[128841]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T02:36:57.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:56 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:57.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:56 smithi146 ceph-mon[128841]: from='osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:36:57.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:56 smithi146 ceph-mon[128841]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:36:57.527 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:36:57.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:36:57.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: osdmap e37: 6 total, 5 up, 6 in 2023-07-20T02:36:57.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:57.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:57.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:57.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi151 2023-07-20T02:36:57.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:57.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:57.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:57.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:57.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi151 to 8584M 2023-07-20T02:36:57.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:57.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:57.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:57.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:57 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:36:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: osdmap e37: 6 total, 5 up, 6 in 2023-07-20T02:36:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi151 2023-07-20T02:36:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:58.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:58.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi151 to 8584M 2023-07-20T02:36:58.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:58.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:58.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:57 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:36:58.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: osdmap e37: 6 total, 5 up, 6 in 2023-07-20T02:36:58.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:58.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:58.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:36:58.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi151 2023-07-20T02:36:58.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:58.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:36:58.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi151 to 8584M 2023-07-20T02:36:58.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:36:58.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:36:58.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:57 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:36:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.710+0000 7f9ccefc2700 1 -- 172.21.15.17:0/1765379346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc81012a0 msgr2=0x7f9cc8107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:58.711 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.710+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/1765379346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc81012a0 0x7f9cc8107b50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000b0d0 tx=0x7f9cc000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:36:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.710+0000 7f9ccefc2700 1 -- 172.21.15.17:0/1765379346 shutdown_connections 2023-07-20T02:36:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.710+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/1765379346 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc81012a0 0x7f9cc8107b50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:58.712 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.710+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/1765379346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc80ff700 0x7f9cc80ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.710+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/1765379346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc8100900 0x7f9cc8100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.710+0000 7f9ccefc2700 1 -- 172.21.15.17:0/1765379346 >> 172.21.15.17:0/1765379346 conn(0x7f9cc80fac80 msgr2=0x7f9cc80fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:36:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 -- 172.21.15.17:0/1765379346 shutdown_connections 2023-07-20T02:36:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 -- 172.21.15.17:0/1765379346 wait complete. 2023-07-20T02:36:58.713 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 Processor -- start 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 -- start start 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc80ff700 0x7f9cc8195480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc8100900 0x7f9cc81959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc81012a0 0x7f9cc819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc819d010 con 0x7f9cc8100900 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc819d150 con 0x7f9cc81012a0 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.711+0000 7f9ccefc2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc819d2e0 con 0x7f9cc80ff700 2023-07-20T02:36:58.714 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc8100900 0x7f9cc81959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc8100900 0x7f9cc81959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:42914/0 (socket says 172.21.15.17:42914) 2023-07-20T02:36:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 -- 172.21.15.17:0/2189523645 learned_addr learned my addr 172.21.15.17:0/2189523645 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:36:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cccd5e700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc80ff700 0x7f9cc8195480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9ccd55f700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc81012a0 0x7f9cc819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 -- 172.21.15.17:0/2189523645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc80ff700 msgr2=0x7f9cc8195480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc80ff700 0x7f9cc8195480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:58.715 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 -- 172.21.15.17:0/2189523645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc81012a0 msgr2=0x7f9cc819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:36:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc81012a0 0x7f9cc819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:36:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 -- 172.21.15.17:0/2189523645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc0009d00 con 0x7f9cc8100900 2023-07-20T02:36:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc7fff700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc8100900 0x7f9cc81959c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400d980 tx=0x7f9cb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:36:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc5ffb700 1 -- 172.21.15.17:0/2189523645 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4009b40 con 0x7f9cc8100900 2023-07-20T02:36:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc8068350 con 0x7f9cc8100900 2023-07-20T02:36:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc5ffb700 1 -- 172.21.15.17:0/2189523645 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cb400b6e0 con 0x7f9cc8100900 2023-07-20T02:36:58.716 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.712+0000 7f9cc5ffb700 1 -- 172.21.15.17:0/2189523645 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400f620 con 0x7f9cc8100900 2023-07-20T02:36:58.717 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.713+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc8068850 con 0x7f9cc8100900 2023-07-20T02:36:58.717 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.713+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc810af10 con 0x7f9cc8100900 2023-07-20T02:36:58.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.714+0000 7f9cc5ffb700 1 -- 172.21.15.17:0/2189523645 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9cb4010460 con 0x7f9cc8100900 2023-07-20T02:36:58.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.715+0000 7f9cc5ffb700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9cb80a1150 0x7f9cb80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:36:58.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.715+0000 7f9cccd5e700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9cb80a1150 0x7f9cb80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:36:58.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.715+0000 7f9cc5ffb700 1 -- 172.21.15.17:0/2189523645 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4566+0+0 (secure 0 0 0) 0x7f9cb40cbc40 con 0x7f9cc8100900 2023-07-20T02:36:58.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.716+0000 7f9cccd5e700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9cb80a1150 0x7f9cb80a3610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8100760 tx=0x7f9cbc007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:36:58.718 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.720+0000 7f9cc5ffb700 1 -- 172.21.15.17:0/2189523645 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9cb4096230 con 0x7f9cc8100900 2023-07-20T02:36:58.927 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:36:58.930+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cc8060bb0 con 0x7f9cb80a1150 2023-07-20T02:36:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:58 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:36:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:58 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:36:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:58 smithi151 ceph-mon[129163]: osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838] boot 2023-07-20T02:36:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:58 smithi151 ceph-mon[129163]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T02:36:59.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:59.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:58 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:36:59.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:58 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:36:59.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:59.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:58 smithi017 ceph-mon[127345]: osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838] boot 2023-07-20T02:36:59.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:58 smithi017 ceph-mon[127345]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T02:36:59.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:59.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:58 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:36:59.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:58 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:36:59.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:36:59.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:58 smithi146 ceph-mon[128841]: osd.5 [v2:172.21.15.151:6808/1633219838,v1:172.21.15.151:6809/1633219838] boot 2023-07-20T02:36:59.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:58 smithi146 ceph-mon[128841]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T02:36:59.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:37:00.058 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:59 smithi017 ceph-mon[127345]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T02:37:00.058 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:59 smithi017 ceph-mon[127345]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:00.058 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:59 smithi017 ceph-mon[127345]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:00.058 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:59 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:00.059 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:59 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:00.059 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:36:59 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:59 smithi151 ceph-mon[129163]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T02:37:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:59 smithi151 ceph-mon[129163]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:59 smithi151 ceph-mon[129163]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:59 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:59 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:36:59 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:00.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:59 smithi146 ceph-mon[128841]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T02:37:00.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:59 smithi146 ceph-mon[128841]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:00.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:59 smithi146 ceph-mon[128841]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:00.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:59 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:00.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:59 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:00.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:36:59 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:00 smithi151 ceph-mon[129163]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T02:37:01.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:00 smithi017 ceph-mon[127345]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T02:37:01.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:00 smithi146 ceph-mon[128841]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T02:37:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:01 smithi151 ceph-mon[129163]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T02:37:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:01 smithi151 ceph-mon[129163]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:01 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2598159020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ced483-d37a-46c1-97ae-288f28b091f7"}]: dispatch 2023-07-20T02:37:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:01 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2598159020' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ced483-d37a-46c1-97ae-288f28b091f7"}]': finished 2023-07-20T02:37:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:01 smithi151 ceph-mon[129163]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T02:37:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:01 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:02.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:01 smithi017 ceph-mon[127345]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T02:37:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:01 smithi017 ceph-mon[127345]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:01 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2598159020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ced483-d37a-46c1-97ae-288f28b091f7"}]: dispatch 2023-07-20T02:37:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:01 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2598159020' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ced483-d37a-46c1-97ae-288f28b091f7"}]': finished 2023-07-20T02:37:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:01 smithi017 ceph-mon[127345]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T02:37:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:01 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:02.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:01 smithi146 ceph-mon[128841]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T02:37:02.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:01 smithi146 ceph-mon[128841]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:02.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:01 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2598159020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ced483-d37a-46c1-97ae-288f28b091f7"}]: dispatch 2023-07-20T02:37:02.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:01 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2598159020' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ced483-d37a-46c1-97ae-288f28b091f7"}]': finished 2023-07-20T02:37:02.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:01 smithi146 ceph-mon[128841]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T02:37:02.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:01 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:02 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1549946260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:03.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:02 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1549946260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:03.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:02 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1549946260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:04.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:03 smithi017 ceph-mon[127345]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-07-20T02:37:04.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:03 smithi146 ceph-mon[128841]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-07-20T02:37:04.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:03 smithi151 ceph-mon[129163]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-07-20T02:37:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:05 smithi151 ceph-mon[129163]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:37:06.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:05 smithi017 ceph-mon[127345]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:37:06.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:05 smithi146 ceph-mon[128841]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:37:08.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:07 smithi017 ceph-mon[127345]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T02:37:08.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:07 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T02:37:08.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:07 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:08.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:07 smithi146 ceph-mon[128841]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T02:37:08.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:07 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T02:37:08.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:07 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:08.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:07 smithi151 ceph-mon[129163]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T02:37:08.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:07 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T02:37:08.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:07 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:09.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:08 smithi017 ceph-mon[127345]: Deploying daemon osd.6 on smithi017 2023-07-20T02:37:09.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:08 smithi146 ceph-mon[128841]: Deploying daemon osd.6 on smithi017 2023-07-20T02:37:09.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:08 smithi151 ceph-mon[129163]: Deploying daemon osd.6 on smithi017 2023-07-20T02:37:10.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:09 smithi017 ceph-mon[127345]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:37:10.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:09 smithi146 ceph-mon[128841]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:37:10.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:09 smithi151 ceph-mon[129163]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:37:12.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:11 smithi017 ceph-mon[127345]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:37:12.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:11 smithi146 ceph-mon[128841]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:37:12.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:11 smithi151 ceph-mon[129163]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:37:13.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:13.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:13.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:12 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:13.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:12 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:13.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:12 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:13.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:12 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:12 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:12 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:13.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:12 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:14.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:14 smithi146 ceph-mon[128841]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T02:37:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:14 smithi151 ceph-mon[129163]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T02:37:14.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:14 smithi017 ceph-mon[127345]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T02:37:15.635 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 6 on host 'smithi017' 2023-07-20T02:37:15.635 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.636+0000 7f9cc5ffb700 1 -- 172.21.15.17:0/2189523645 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9cc8060bb0 con 0x7f9cb80a1150 2023-07-20T02:37:15.637 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9cb80a1150 msgr2=0x7f9cb80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:15.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9cb80a1150 0x7f9cb80a3610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8100760 tx=0x7f9cbc007400 comp rx=0 tx=0).stop 2023-07-20T02:37:15.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc8100900 msgr2=0x7f9cc81959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:15.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc8100900 0x7f9cc81959c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400d980 tx=0x7f9cb400dc90 comp rx=0 tx=0).stop 2023-07-20T02:37:15.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 shutdown_connections 2023-07-20T02:37:15.638 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9cb80a1150 0x7f9cb80a3610 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:15.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc81012a0 0x7f9cc819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:15.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc80ff700 0x7f9cc8195480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:15.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 --2- 172.21.15.17:0/2189523645 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9cc8100900 0x7f9cc81959c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:15.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 >> 172.21.15.17:0/2189523645 conn(0x7f9cc80fac80 msgr2=0x7f9cc80fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:37:15.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 shutdown_connections 2023-07-20T02:37:15.639 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:37:15.639+0000 7f9ccefc2700 1 -- 172.21.15.17:0/2189523645 wait complete. 2023-07-20T02:37:16.122 DEBUG:teuthology.orchestra.run.smithi017:osd.6> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.6.service 2023-07-20T02:37:16.124 INFO:tasks.cephadm:Deploying osd.7 on smithi146 with /dev/vg_nvme/lv_2... 2023-07-20T02:37:16.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T02:37:16.170 INFO:journalctl@ceph.osd.6.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:37:16.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:16 smithi151 ceph-mon[129163]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:16.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:16 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:16 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:16 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:16 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:16 smithi017 ceph-mon[127345]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:16.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:16 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:16 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:16 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:16 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:16 smithi146 ceph-mon[128841]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:16.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:16 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:16 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:16 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:16 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:16.704 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:37:16.896 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:37:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:37:16.470+0000 7fafe5cc0640 -1 osd.6 0 log_to_monitors true 2023-07-20T02:37:17.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:17 smithi017 ceph-mon[127345]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:37:17.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:17 smithi146 ceph-mon[128841]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:37:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:17 smithi151 ceph-mon[129163]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:37:18.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:18 smithi146 ceph-mon[128841]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:18.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:18 smithi146 ceph-mon[128841]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T02:37:18.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:18 smithi146 ceph-mon[128841]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T02:37:18.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:18 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:18.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:18 smithi146 ceph-mon[128841]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:37:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:18 smithi151 ceph-mon[129163]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:18 smithi151 ceph-mon[129163]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T02:37:18.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:18 smithi151 ceph-mon[129163]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T02:37:18.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:18 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:18.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:18 smithi151 ceph-mon[129163]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:37:18.406 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:37:18 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:37:18.091+0000 7fafdab37700 -1 osd.6 0 waiting for initial osdmap 2023-07-20T02:37:18.406 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:37:18 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:37:18.103+0000 7fafd712e700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:37:18.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:18 smithi017 ceph-mon[127345]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:18.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:18 smithi017 ceph-mon[127345]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T02:37:18.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:18 smithi017 ceph-mon[127345]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T02:37:18.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:18 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:18.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:18 smithi017 ceph-mon[127345]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:37:18.466 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T02:37:18.466 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T02:37:18.466 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2023-07-20T02:37:18.467 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2023-07-20T02:37:18.467 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111055 s, 944 MB/s 2023-07-20T02:37:18.467 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2023-07-20T02:37:18.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2023-07-20T02:37:19.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:37:19.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T02:37:19.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:19.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:19.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi017 2023-07-20T02:37:19.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi017 to 5724M 2023-07-20T02:37:19.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:19.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:19.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:19.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:19.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:19.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi017 2023-07-20T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi017 to 5724M 2023-07-20T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:37:19.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T02:37:19.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:19.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:19.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi017 2023-07-20T02:37:19.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:19.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi017 to 5724M 2023-07-20T02:37:19.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:19.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:19.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:19.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:19.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:19.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:19.506 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:37:20.340 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:20 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:37:20.340 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:20 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:37:20.341 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:20 smithi146 ceph-mon[128841]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:20.341 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:20 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:20.341 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:20 smithi146 ceph-mon[128841]: osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822] boot 2023-07-20T02:37:20.341 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:20 smithi146 ceph-mon[128841]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T02:37:20.341 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:20 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:20.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:20 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:37:20.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:20 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:37:20.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:20 smithi151 ceph-mon[129163]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:20.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:20 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:20.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:20 smithi151 ceph-mon[129163]: osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822] boot 2023-07-20T02:37:20.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:20 smithi151 ceph-mon[129163]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T02:37:20.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:20 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:20.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:20 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:37:20.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:20 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:37:20.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:20 smithi017 ceph-mon[127345]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:37:20.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:20 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:20.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:20 smithi017 ceph-mon[127345]: osd.6 [v2:172.21.15.17:6818/3807389822,v1:172.21.15.17:6819/3807389822] boot 2023-07-20T02:37:20.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:20 smithi017 ceph-mon[127345]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T02:37:20.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:20 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:37:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 -- 172.21.15.146:0/1201226734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 msgr2=0x7fd28c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 --2- 172.21.15.146:0/1201226734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c0ffb20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0099c0 tx=0x7fd27c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:37:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 -- 172.21.15.146:0/1201226734 shutdown_connections 2023-07-20T02:37:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 --2- 172.21.15.146:0/1201226734 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd28c1012a0 0x7fd28c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 --2- 172.21.15.146:0/1201226734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c0ffb20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 --2- 172.21.15.146:0/1201226734 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd28c100900 0x7fd28c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 -- 172.21.15.146:0/1201226734 >> 172.21.15.146:0/1201226734 conn(0x7fd28c0fac80 msgr2=0x7fd28c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:37:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 -- 172.21.15.146:0/1201226734 shutdown_connections 2023-07-20T02:37:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.827+0000 7fd290a56700 1 -- 172.21.15.146:0/1201226734 wait complete. 2023-07-20T02:37:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 Processor -- start 2023-07-20T02:37:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 -- start start 2023-07-20T02:37:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c194270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd28c100900 0x7fd28c1947b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd28c1012a0 0x7fd28c19b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c19be00 con 0x7fd28c1012a0 2023-07-20T02:37:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c19bf70 con 0x7fd28c0ff700 2023-07-20T02:37:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd290a56700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c19c130 con 0x7fd28c100900 2023-07-20T02:37:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c194270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c194270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:52324/0 (socket says 172.21.15.146:52324) 2023-07-20T02:37:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 -- 172.21.15.146:0/2231831751 learned_addr learned my addr 172.21.15.146:0/2231831751 (peer_addr_for_me v2:172.21.15.146:0/0) 2023-07-20T02:37:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd289d9b700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd28c100900 0x7fd28c1947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28ad9d700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd28c1012a0 0x7fd28c19b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 -- 172.21.15.146:0/2231831751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd28c100900 msgr2=0x7fd28c1947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd28c100900 0x7fd28c1947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 -- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd28c1012a0 msgr2=0x7fd28c19b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd28c1012a0 0x7fd28c19b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 -- 172.21.15.146:0/2231831751 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c0095f0 con 0x7fd28c0ff700 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28ad9d700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd28c1012a0 0x7fd28c19b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:37:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.828+0000 7fd28a59c700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c194270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009990 tx=0x7fd27c003af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:37:20.831 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.829+0000 7fd27b7fe700 1 -- 172.21.15.146:0/2231831751 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c01d070 con 0x7fd28c0ff700 2023-07-20T02:37:20.831 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.829+0000 7fd27b7fe700 1 -- 172.21.15.146:0/2231831751 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd27c004640 con 0x7fd28c0ff700 2023-07-20T02:37:20.831 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.829+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c19c3b0 con 0x7fd28c0ff700 2023-07-20T02:37:20.831 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.829+0000 7fd27b7fe700 1 -- 172.21.15.146:0/2231831751 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c00f460 con 0x7fd28c0ff700 2023-07-20T02:37:20.831 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.829+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c19c850 con 0x7fd28c0ff700 2023-07-20T02:37:20.831 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.830+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c10af80 con 0x7fd28c0ff700 2023-07-20T02:37:20.832 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.832+0000 7fd27b7fe700 1 -- 172.21.15.146:0/2231831751 <== mon.2 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd27c003fa0 con 0x7fd28c0ff700 2023-07-20T02:37:20.832 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.832+0000 7fd27b7fe700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd2740a1150 0x7fd2740a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:20.832 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.832+0000 7fd27b7fe700 1 -- 172.21.15.146:0/2231831751 <== mon.2 v2:172.21.15.146:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fd27c0cd7b0 con 0x7fd28c0ff700 2023-07-20T02:37:20.832 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.832+0000 7fd289d9b700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd2740a1150 0x7fd2740a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:20.833 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.833+0000 7fd289d9b700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd2740a1150 0x7fd2740a3610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd28000ac50 tx=0x7fd28000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:37:20.833 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:20.836+0000 7fd27b7fe700 1 -- 172.21.15.146:0/2231831751 <== mon.2 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd27c097ca0 con 0x7fd28c0ff700 2023-07-20T02:37:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:21.031+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd28c0692d0 con 0x7fd2740a1150 2023-07-20T02:37:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:21 smithi151 ceph-mon[129163]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T02:37:21.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:21.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:21.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:21.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:21 smithi017 ceph-mon[127345]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T02:37:21.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:21.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:21.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:21.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:21 smithi146 ceph-mon[128841]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T02:37:21.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:21.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:21.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:22.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:22 smithi146 ceph-mon[128841]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:22.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:22 smithi146 ceph-mon[128841]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:22.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:22 smithi017 ceph-mon[127345]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:22.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:22 smithi017 ceph-mon[127345]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:22.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:22 smithi151 ceph-mon[129163]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:22.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:22 smithi151 ceph-mon[129163]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:24.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:24 smithi017 ceph-mon[127345]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:24.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:24 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/2332625849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]: dispatch 2023-07-20T02:37:24.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:24 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]: dispatch 2023-07-20T02:37:24.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:24 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]': finished 2023-07-20T02:37:24.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:24 smithi017 ceph-mon[127345]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T02:37:24.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:24 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:24.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:24 smithi146 ceph-mon[128841]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:24.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:24 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/2332625849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]: dispatch 2023-07-20T02:37:24.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:24 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]: dispatch 2023-07-20T02:37:24.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:24 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]': finished 2023-07-20T02:37:24.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:24 smithi146 ceph-mon[128841]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T02:37:24.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:24 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:24.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:24 smithi151 ceph-mon[129163]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:24.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:24 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/2332625849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]: dispatch 2023-07-20T02:37:24.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:24 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]: dispatch 2023-07-20T02:37:24.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:24 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26199bbc-d82f-4e97-9218-0058426ef412"}]': finished 2023-07-20T02:37:24.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:24 smithi151 ceph-mon[129163]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T02:37:24.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:24 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:25.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:25 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/2473431206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:25.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:25 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/2473431206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:25 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/2473431206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:26.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:26 smithi017 ceph-mon[127345]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:26.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:26 smithi146 ceph-mon[128841]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:26.615 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:26 smithi151 ceph-mon[129163]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:28.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:28 smithi146 ceph-mon[128841]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:28.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:28 smithi017 ceph-mon[127345]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:28 smithi151 ceph-mon[129163]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:30.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:30 smithi017 ceph-mon[127345]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:30.565 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:30 smithi146 ceph-mon[128841]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:30 smithi151 ceph-mon[129163]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:31.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T02:37:31.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:31.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:31 smithi017 ceph-mon[127345]: Deploying daemon osd.7 on smithi146 2023-07-20T02:37:31.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T02:37:31.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:31.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:31 smithi146 ceph-mon[128841]: Deploying daemon osd.7 on smithi146 2023-07-20T02:37:31.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T02:37:31.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:31.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:31 smithi151 ceph-mon[129163]: Deploying daemon osd.7 on smithi146 2023-07-20T02:37:32.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:32 smithi146 ceph-mon[128841]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:32.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:32 smithi017 ceph-mon[127345]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:32 smithi151 ceph-mon[129163]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:34.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:34 smithi146 ceph-mon[128841]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:34.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:34 smithi017 ceph-mon[127345]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:34.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:34 smithi151 ceph-mon[129163]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:36.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:36 smithi146 ceph-mon[128841]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:36.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:36.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:36.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:36 smithi151 ceph-mon[129163]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:36.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:36 smithi017 ceph-mon[127345]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:36.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:36.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:36.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:38.651 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.652+0000 7fd27b7fe700 1 -- 172.21.15.146:0/2231831751 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd28c0692d0 con 0x7fd2740a1150 2023-07-20T02:37:38.651 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 7 on host 'smithi146' 2023-07-20T02:37:38.653 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd2740a1150 msgr2=0x7fd2740a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:38.653 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd2740a1150 0x7fd2740a3610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd28000ac50 tx=0x7fd28000a380 comp rx=0 tx=0).stop 2023-07-20T02:37:38.654 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 msgr2=0x7fd28c194270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:38.654 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c194270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009990 tx=0x7fd27c003af0 comp rx=0 tx=0).stop 2023-07-20T02:37:38.654 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 shutdown_connections 2023-07-20T02:37:38.654 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd2740a1150 0x7fd2740a3610 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:38.655 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd28c1012a0 0x7fd28c19b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:38.655 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd28c0ff700 0x7fd28c194270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:38.655 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 --2- 172.21.15.146:0/2231831751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd28c100900 0x7fd28c1947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:38.655 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 >> 172.21.15.146:0/2231831751 conn(0x7fd28c0fac80 msgr2=0x7fd28c068bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:37:38.655 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 shutdown_connections 2023-07-20T02:37:38.655 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:37:38.655+0000 7fd290a56700 1 -- 172.21.15.146:0/2231831751 wait complete. 2023-07-20T02:37:38.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:38 smithi146 ceph-mon[128841]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:38.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:38.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:38.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:38 smithi151 ceph-mon[129163]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:38.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:38.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:38.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:38 smithi017 ceph-mon[127345]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:38.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:38.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:39.201 DEBUG:teuthology.orchestra.run.smithi146:osd.7> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.7.service 2023-07-20T02:37:39.203 INFO:tasks.cephadm:Deploying osd.8 on smithi151 with /dev/vg_nvme/lv_2... 2023-07-20T02:37:39.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T02:37:39.233 INFO:journalctl@ceph.osd.7.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:37:39.756 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:37:39 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:37:39.425+0000 7eff09e2f640 -1 osd.7 0 log_to_monitors true 2023-07-20T02:37:39.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:39 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:39.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:39 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:39.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:39 smithi146 ceph-mon[128841]: from='osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:37:39.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:39 smithi146 ceph-mon[128841]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:37:39.779 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:37:39.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:39 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:39.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:39 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:39.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:39 smithi017 ceph-mon[127345]: from='osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:37:39.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:39 smithi017 ceph-mon[127345]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:37:40.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:39 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:40.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:39 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:40.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:39 smithi151 ceph-mon[129163]: from='osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:37:40.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:39 smithi151 ceph-mon[129163]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:37:40.724 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:37:40.642+0000 7efefeca6700 -1 osd.7 0 waiting for initial osdmap 2023-07-20T02:37:40.724 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:37:40.652+0000 7efef8297700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:37:40.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-mon[128841]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:40.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-mon[128841]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T02:37:40.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-mon[128841]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T02:37:40.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:40.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-mon[128841]: from='osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:37:40.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:40 smithi146 ceph-mon[128841]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:37:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:40 smithi151 ceph-mon[129163]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:40 smithi151 ceph-mon[129163]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T02:37:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:40 smithi151 ceph-mon[129163]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T02:37:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:40 smithi151 ceph-mon[129163]: from='osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:37:40.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:40 smithi151 ceph-mon[129163]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:37:40.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:40 smithi017 ceph-mon[127345]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:40.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:40 smithi017 ceph-mon[127345]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T02:37:40.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:40 smithi017 ceph-mon[127345]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T02:37:40.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:40.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:40 smithi017 ceph-mon[127345]: from='osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:37:40.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:40 smithi017 ceph-mon[127345]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:37:41.372 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T02:37:41.373 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T02:37:41.373 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2023-07-20T02:37:41.373 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2023-07-20T02:37:41.373 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124471 s, 842 MB/s 2023-07-20T02:37:41.373 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2023-07-20T02:37:41.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:41 smithi151 ceph-mon[129163]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:37:41.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:41 smithi151 ceph-mon[129163]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T02:37:41.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:41.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:41.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi151:vg_nvme/lv_2 2023-07-20T02:37:41.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:41 smithi017 ceph-mon[127345]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:37:41.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:41 smithi017 ceph-mon[127345]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T02:37:41.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:41.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:42.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:41 smithi146 ceph-mon[128841]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:37:42.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:41 smithi146 ceph-mon[128841]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T02:37:42.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:42.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:42.284 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:37:42.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:37:42.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:37:42.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:42.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:42.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi146 2023-07-20T02:37:42.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673] boot 2023-07-20T02:37:42.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T02:37:42.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:42.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:42.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:42.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:42.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi146 to 5722M 2023-07-20T02:37:42.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:42.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:42.813 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.813 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:42.813 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:42.813 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:42.813 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:37:42.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:37:42.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:42.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:42.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi146 2023-07-20T02:37:42.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673] boot 2023-07-20T02:37:42.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T02:37:42.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:42.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:42.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:42.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:42.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi146 to 5722M 2023-07-20T02:37:42.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:43.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:37:43.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:37:43.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:37:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi146 2023-07-20T02:37:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: osd.7 [v2:172.21.15.146:6816/708669673,v1:172.21.15.146:6817/708669673] boot 2023-07-20T02:37:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T02:37:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:37:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:37:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi146 to 5722M 2023-07-20T02:37:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:43.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:43.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:43.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:43.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:37:43.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:43.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 -- 172.21.15.151:0/600543480 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe08100900 msgr2=0x7efe08100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:43.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 --2- 172.21.15.151:0/600543480 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe08100900 0x7efe08100d60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009990 tx=0x7efdf0009ca0 comp rx=0 tx=0).stop 2023-07-20T02:37:43.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 -- 172.21.15.151:0/600543480 shutdown_connections 2023-07-20T02:37:43.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 --2- 172.21.15.151:0/600543480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe081012a0 0x7efe08107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:43.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 --2- 172.21.15.151:0/600543480 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe080ff700 0x7efe080ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:43.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 --2- 172.21.15.151:0/600543480 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe08100900 0x7efe08100d60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:43.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 -- 172.21.15.151:0/600543480 >> 172.21.15.151:0/600543480 conn(0x7efe080fac80 msgr2=0x7efe080fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:37:43.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 -- 172.21.15.151:0/600543480 shutdown_connections 2023-07-20T02:37:43.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.516+0000 7efe0c967700 1 -- 172.21.15.151:0/600543480 wait complete. 2023-07-20T02:37:43.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 Processor -- start 2023-07-20T02:37:43.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 -- start start 2023-07-20T02:37:43.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe080ff700 0x7efe08195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:43.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe08100900 0x7efe081958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:43.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe081012a0 0x7efe0819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:43.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0819cf00 con 0x7efe080ff700 2023-07-20T02:37:43.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0819d040 con 0x7efe08100900 2023-07-20T02:37:43.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0c967700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0819d1d0 con 0x7efe081012a0 2023-07-20T02:37:43.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe06d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe081012a0 0x7efe0819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:43.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe05d9b700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe08100900 0x7efe081958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:43.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe06d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe081012a0 0x7efe0819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40950/0 (socket says 172.21.15.151:40950) 2023-07-20T02:37:43.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe0659c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe080ff700 0x7efe08195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:43.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.517+0000 7efe06d9d700 1 -- 172.21.15.151:0/1974610970 learned_addr learned my addr 172.21.15.151:0/1974610970 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:37:43.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe06d9d700 1 -- 172.21.15.151:0/1974610970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe08100900 msgr2=0x7efe081958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:43.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe06d9d700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe08100900 0x7efe081958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:43.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe06d9d700 1 -- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe080ff700 msgr2=0x7efe08195370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:37:43.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe06d9d700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe080ff700 0x7efe08195370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:37:43.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe06d9d700 1 -- 172.21.15.151:0/1974610970 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00095f0 con 0x7efe081012a0 2023-07-20T02:37:43.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe06d9d700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe081012a0 0x7efe0819c930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efe0000e970 tx=0x7efe0000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:37:43.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efdff7fe700 1 -- 172.21.15.151:0/1974610970 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe000097b0 con 0x7efe081012a0 2023-07-20T02:37:43.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0819d4b0 con 0x7efe081012a0 2023-07-20T02:37:43.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efdff7fe700 1 -- 172.21.15.151:0/1974610970 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe00005020 con 0x7efe081012a0 2023-07-20T02:37:43.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efdff7fe700 1 -- 172.21.15.151:0/1974610970 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0000cc70 con 0x7efe081012a0 2023-07-20T02:37:43.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.518+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0819d9b0 con 0x7efe081012a0 2023-07-20T02:37:43.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.519+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0810af00 con 0x7efe081012a0 2023-07-20T02:37:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.520+0000 7efdff7fe700 1 -- 172.21.15.151:0/1974610970 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efe00005470 con 0x7efe081012a0 2023-07-20T02:37:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.521+0000 7efdff7fe700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7efdf40a1150 0x7efdf40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:37:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.521+0000 7efdff7fe700 1 -- 172.21.15.151:0/1974610970 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7efe00014070 con 0x7efe081012a0 2023-07-20T02:37:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.521+0000 7efe0659c700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7efdf40a1150 0x7efdf40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:37:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.522+0000 7efe0659c700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7efdf40a1150 0x7efdf40a3610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009550 tx=0x7efdf80091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:37:43.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.524+0000 7efdff7fe700 1 -- 172.21.15.151:0/1974610970 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efe000963b0 con 0x7efe081012a0 2023-07-20T02:37:43.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:37:43.727+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efe0819e0b0 con 0x7efdf40a1150 2023-07-20T02:37:43.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:43 smithi017 ceph-mon[127345]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T02:37:44.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:43 smithi146 ceph-mon[128841]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T02:37:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:43 smithi151 ceph-mon[129163]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T02:37:44.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:44 smithi151 ceph-mon[129163]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:44.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:44 smithi151 ceph-mon[129163]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T02:37:44.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:44 smithi151 ceph-mon[129163]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:44.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:44.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:44.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:44.952 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:37:44 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:37:44.854+0000 7f85ec148700 -1 calc_pg_upmaps abort due to max <= 0 2023-07-20T02:37:44.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:44 smithi017 ceph-mon[127345]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:44.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:44 smithi017 ceph-mon[127345]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T02:37:44.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:44 smithi017 ceph-mon[127345]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:44.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:44.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:44.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:45.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:44 smithi146 ceph-mon[128841]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:45.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:44 smithi146 ceph-mon[128841]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T02:37:45.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:44 smithi146 ceph-mon[128841]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:37:45.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:37:45.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:37:45.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:45.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:45 smithi017 ceph-mon[127345]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T02:37:46.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:45 smithi146 ceph-mon[128841]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T02:37:46.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:45 smithi151 ceph-mon[129163]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T02:37:46.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:46 smithi017 ceph-mon[127345]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:46.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:46 smithi017 ceph-mon[127345]: osdmap e54: 8 total, 8 up, 8 in 2023-07-20T02:37:46.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:46 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/1434728238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]: dispatch 2023-07-20T02:37:46.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:46 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]: dispatch 2023-07-20T02:37:46.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:46 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]': finished 2023-07-20T02:37:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:46 smithi017 ceph-mon[127345]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T02:37:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:46 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:37:47.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:46 smithi146 ceph-mon[128841]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:47.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:46 smithi146 ceph-mon[128841]: osdmap e54: 8 total, 8 up, 8 in 2023-07-20T02:37:47.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:46 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/1434728238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]: dispatch 2023-07-20T02:37:47.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:46 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]: dispatch 2023-07-20T02:37:47.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:46 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]': finished 2023-07-20T02:37:47.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:46 smithi146 ceph-mon[128841]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T02:37:47.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:46 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:37:47.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:46 smithi151 ceph-mon[129163]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:47.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:46 smithi151 ceph-mon[129163]: osdmap e54: 8 total, 8 up, 8 in 2023-07-20T02:37:47.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:46 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/1434728238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]: dispatch 2023-07-20T02:37:47.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:46 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]: dispatch 2023-07-20T02:37:47.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:46 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1a33778-9ab5-45db-90ff-644794ee1cd8"}]': finished 2023-07-20T02:37:47.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:46 smithi151 ceph-mon[129163]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T02:37:47.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:46 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:37:47.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:47 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/2456566579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:48.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:47 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/2456566579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:48.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:47 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/2456566579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:37:48.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:48 smithi017 ceph-mon[127345]: pgmap v166: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:49.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:48 smithi146 ceph-mon[128841]: pgmap v166: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:49.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:48 smithi151 ceph-mon[129163]: pgmap v166: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:50.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:50 smithi017 ceph-mon[127345]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:51.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:50 smithi146 ceph-mon[128841]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:51.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:50 smithi151 ceph-mon[129163]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:52.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:52 smithi017 ceph-mon[127345]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:53.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:52 smithi146 ceph-mon[128841]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:53.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:52 smithi151 ceph-mon[129163]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:53.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T02:37:53.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:53.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:53 smithi017 ceph-mon[127345]: Deploying daemon osd.8 on smithi151 2023-07-20T02:37:53.992 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T02:37:53.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:53.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:53 smithi151 ceph-mon[129163]: Deploying daemon osd.8 on smithi151 2023-07-20T02:37:54.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T02:37:54.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:37:54.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:53 smithi146 ceph-mon[128841]: Deploying daemon osd.8 on smithi151 2023-07-20T02:37:54.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:54 smithi151 ceph-mon[129163]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:54.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:54 smithi017 ceph-mon[127345]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:55.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:54 smithi146 ceph-mon[128841]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:56.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:55 smithi151 ceph-mon[129163]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:56.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:55 smithi017 ceph-mon[127345]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:56.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:55 smithi146 ceph-mon[128841]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:58.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:58 smithi151 ceph-mon[129163]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:58.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:58.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:58.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:37:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:58.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:58 smithi146 ceph-mon[128841]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:58.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:58.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:58.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:37:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:37:58.702 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:58 smithi017 ceph-mon[127345]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:37:58.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:58.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:37:58.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:37:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:00.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.339+0000 7efdff7fe700 1 -- 172.21.15.151:0/1974610970 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efe0819e0b0 con 0x7efdf40a1150 2023-07-20T02:38:00.340 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 8 on host 'smithi151' 2023-07-20T02:38:00.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7efdf40a1150 msgr2=0x7efdf40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:00.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7efdf40a1150 0x7efdf40a3610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009550 tx=0x7efdf80091b0 comp rx=0 tx=0).stop 2023-07-20T02:38:00.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe081012a0 msgr2=0x7efe0819c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:00.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe081012a0 0x7efe0819c930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efe0000e970 tx=0x7efe0000ec80 comp rx=0 tx=0).stop 2023-07-20T02:38:00.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 shutdown_connections 2023-07-20T02:38:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7efdf40a1150 0x7efdf40a3610 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe081012a0 0x7efe0819c930 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7efe080ff700 0x7efe08195370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 --2- 172.21.15.151:0/1974610970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe08100900 0x7efe081958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 >> 172.21.15.151:0/1974610970 conn(0x7efe080fac80 msgr2=0x7efe080fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:38:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 shutdown_connections 2023-07-20T02:38:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:00.342+0000 7efe0c967700 1 -- 172.21.15.151:0/1974610970 wait complete. 2023-07-20T02:38:00.764 DEBUG:teuthology.orchestra.run.smithi151:osd.8> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.8.service 2023-07-20T02:38:00.767 INFO:tasks.cephadm:Deploying osd.9 on smithi017 with /dev/vg_nvme/lv_1... 2023-07-20T02:38:00.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T02:38:00.799 INFO:journalctl@ceph.osd.8.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:38:00.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:00.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:00 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:01.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:00 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:01.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:01.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:01.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:01.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:01.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:00 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:01.342 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:38:01.520 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:38:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:38:01.188+0000 7f7ec6476640 -1 osd.8 0 log_to_monitors true 2023-07-20T02:38:01.777 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:01 smithi017 ceph-mon[127345]: from='osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:38:01.777 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:01 smithi017 ceph-mon[127345]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:38:01.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:01 smithi151 ceph-mon[129163]: from='osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:38:01.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:01 smithi151 ceph-mon[129163]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:38:02.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:01 smithi146 ceph-mon[128841]: from='osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:38:02.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:01 smithi146 ceph-mon[128841]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:38:02.861 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T02:38:02.861 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T02:38:02.862 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2023-07-20T02:38:02.862 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2023-07-20T02:38:02.862 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00800392 s, 1.3 GB/s 2023-07-20T02:38:02.862 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2023-07-20T02:38:03.019 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:02 smithi017 ceph-mon[127345]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:03.019 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:02 smithi017 ceph-mon[127345]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T02:38:03.020 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:02 smithi017 ceph-mon[127345]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T02:38:03.020 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:02 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:03.020 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:02 smithi017 ceph-mon[127345]: from='osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:38:03.020 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:02 smithi017 ceph-mon[127345]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:38:03.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:02 smithi146 ceph-mon[128841]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:03.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:02 smithi146 ceph-mon[128841]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T02:38:03.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:02 smithi146 ceph-mon[128841]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T02:38:03.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:02 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:03.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:02 smithi146 ceph-mon[128841]: from='osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:38:03.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:02 smithi146 ceph-mon[128841]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:38:03.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-mon[129163]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:03.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-mon[129163]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T02:38:03.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-mon[129163]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T02:38:03.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:03.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-mon[129163]: from='osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:38:03.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-mon[129163]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:38:03.112 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:38:02.715+0000 7f7ebb2ed700 -1 osd.8 0 waiting for initial osdmap 2023-07-20T02:38:03.113 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:38:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:38:02.727+0000 7f7eb58e0700 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:38:03.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi017:vg_nvme/lv_1 2023-07-20T02:38:03.867 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:38:04.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:38:04.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: osdmap e57: 9 total, 8 up, 9 in 2023-07-20T02:38:04.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:04.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:04.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:04.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:04.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:38:04.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: osdmap e57: 9 total, 8 up, 9 in 2023-07-20T02:38:04.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:04.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:04.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:04.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:04.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:38:04.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: osdmap e57: 9 total, 8 up, 9 in 2023-07-20T02:38:04.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:04.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:04.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:04.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:04.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:04.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:04.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:05.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 -- 172.21.15.17:0/2933082164 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff200100900 msgr2=0x7ff200100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:05.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 --2- 172.21.15.17:0/2933082164 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff200100900 0x7ff200100d60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0099c0 tx=0x7ff1fc009cd0 comp rx=0 tx=0).stop 2023-07-20T02:38:05.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 -- 172.21.15.17:0/2933082164 shutdown_connections 2023-07-20T02:38:05.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 --2- 172.21.15.17:0/2933082164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2001012a0 0x7ff200107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:05.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 --2- 172.21.15.17:0/2933082164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2000ff700 0x7ff2000ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:05.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 --2- 172.21.15.17:0/2933082164 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff200100900 0x7ff200100d60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:05.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 -- 172.21.15.17:0/2933082164 >> 172.21.15.17:0/2933082164 conn(0x7ff2000fac80 msgr2=0x7ff2000fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:38:05.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 -- 172.21.15.17:0/2933082164 shutdown_connections 2023-07-20T02:38:05.038 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.036+0000 7ff207d16700 1 -- 172.21.15.17:0/2933082164 wait complete. 2023-07-20T02:38:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 Processor -- start 2023-07-20T02:38:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 -- start start 2023-07-20T02:38:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2000ff700 0x7ff200068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff200100900 0x7ff20019b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2001012a0 0x7ff20019d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20019e020 con 0x7ff2000ff700 2023-07-20T02:38:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20019e190 con 0x7ff2001012a0 2023-07-20T02:38:05.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff207d16700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20019e350 con 0x7ff200100900 2023-07-20T02:38:05.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff2052b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff200100900 0x7ff20019b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:05.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff205ab2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2000ff700 0x7ff200068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:05.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff2052b1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff200100900 0x7ff20019b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.17:44114/0 (socket says 172.21.15.17:44114) 2023-07-20T02:38:05.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff2052b1700 1 -- 172.21.15.17:0/3050680132 learned_addr learned my addr 172.21.15.17:0/3050680132 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:38:05.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.037+0000 7ff2062b3700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2001012a0 0x7ff20019d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff205ab2700 1 -- 172.21.15.17:0/3050680132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff200100900 msgr2=0x7ff20019b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff205ab2700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff200100900 0x7ff20019b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff205ab2700 1 -- 172.21.15.17:0/3050680132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2001012a0 msgr2=0x7ff20019d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff205ab2700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2001012a0 0x7ff20019d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff205ab2700 1 -- 172.21.15.17:0/3050680132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0095f0 con 0x7ff2000ff700 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff2062b3700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2001012a0 0x7ff20019d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff205ab2700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2000ff700 0x7ff200068d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000d740 tx=0x7ff1f000da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:38:05.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff1f6ffd700 1 -- 172.21.15.17:0/3050680132 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0004220 con 0x7ff2000ff700 2023-07-20T02:38:05.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff1f6ffd700 1 -- 172.21.15.17:0/3050680132 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1f0004380 con 0x7ff2000ff700 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20019e630 con 0x7ff2000ff700 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff1f6ffd700 1 -- 172.21.15.17:0/3050680132 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0003cf0 con 0x7ff2000ff700 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.038+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20019eb80 con 0x7ff2000ff700 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.039+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff200106d00 con 0x7ff2000ff700 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.041+0000 7ff1f6ffd700 1 -- 172.21.15.17:0/3050680132 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff1f000f690 con 0x7ff2000ff700 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.041+0000 7ff1f6ffd700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff1ec0a11d0 0x7ff1ec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.042+0000 7ff2052b1700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff1ec0a11d0 0x7ff1ec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:05.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.042+0000 7ff1f6ffd700 1 -- 172.21.15.17:0/3050680132 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5858+0+0 (secure 0 0 0) 0x7ff1f00cbaa0 con 0x7ff2000ff700 2023-07-20T02:38:05.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.042+0000 7ff2052b1700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff1ec0a11d0 0x7ff1ec0a3690 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff200069310 tx=0x7ff1fc009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:38:05.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.045+0000 7ff1f6ffd700 1 -- 172.21.15.17:0/3050680132 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff1f0095c00 con 0x7ff2000ff700 2023-07-20T02:38:05.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:38:05.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:38:05.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:05.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi151 2023-07-20T02:38:05.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi151 to 5722M 2023-07-20T02:38:05.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:05.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088] boot 2023-07-20T02:38:05.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T02:38:05.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:04 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:38:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:38:05.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:05.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi151 2023-07-20T02:38:05.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi151 to 5722M 2023-07-20T02:38:05.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:05.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088] boot 2023-07-20T02:38:05.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T02:38:05.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:04 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:05.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:38:05.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:38:05.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:38:05.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi151 2023-07-20T02:38:05.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi151 to 5722M 2023-07-20T02:38:05.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:05.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: osd.8 [v2:172.21.15.151:6816/2042667088,v1:172.21.15.151:6817/2042667088] boot 2023-07-20T02:38:05.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T02:38:05.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:04 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:38:05.251 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:05.251+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff200060bb0 con 0x7ff1ec0a11d0 2023-07-20T02:38:06.049 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:05 smithi017 ceph-mon[127345]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T02:38:06.049 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:06.049 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:06.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:05 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:06.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:05 smithi146 ceph-mon[128841]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T02:38:06.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:06.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:06.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:05 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:06.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:05 smithi151 ceph-mon[129163]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T02:38:06.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:06.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:06.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:05 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:07.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:06 smithi146 ceph-mon[128841]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:07.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:06 smithi146 ceph-mon[128841]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:07.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:06 smithi146 ceph-mon[128841]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T02:38:07.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:06 smithi151 ceph-mon[129163]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:07.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:06 smithi151 ceph-mon[129163]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:07.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:06 smithi151 ceph-mon[129163]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T02:38:07.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:06 smithi017 ceph-mon[127345]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:07.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:06 smithi017 ceph-mon[127345]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:07.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:06 smithi017 ceph-mon[127345]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T02:38:08.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:07 smithi151 ceph-mon[129163]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T02:38:08.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:07 smithi151 ceph-mon[129163]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:08.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:07 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/4262149982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "496d04b3-61a4-4405-aa52-5590aacf7129"}]: dispatch 2023-07-20T02:38:08.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:07 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/4262149982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "496d04b3-61a4-4405-aa52-5590aacf7129"}]': finished 2023-07-20T02:38:08.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:07 smithi151 ceph-mon[129163]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T02:38:08.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:07 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:08.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:07 smithi017 ceph-mon[127345]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T02:38:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:07 smithi017 ceph-mon[127345]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:07 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4262149982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "496d04b3-61a4-4405-aa52-5590aacf7129"}]: dispatch 2023-07-20T02:38:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:07 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4262149982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "496d04b3-61a4-4405-aa52-5590aacf7129"}]': finished 2023-07-20T02:38:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:07 smithi017 ceph-mon[127345]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T02:38:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:07 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:08.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:07 smithi146 ceph-mon[128841]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T02:38:08.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:07 smithi146 ceph-mon[128841]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:08.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:07 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/4262149982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "496d04b3-61a4-4405-aa52-5590aacf7129"}]: dispatch 2023-07-20T02:38:08.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:07 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/4262149982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "496d04b3-61a4-4405-aa52-5590aacf7129"}]': finished 2023-07-20T02:38:08.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:07 smithi146 ceph-mon[128841]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T02:38:08.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:07 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:09.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:08 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1164268404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:09.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:08 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1164268404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:09.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:08 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1164268404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:10.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:09 smithi151 ceph-mon[129163]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2023-07-20T02:38:10.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:09 smithi017 ceph-mon[127345]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2023-07-20T02:38:10.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:09 smithi146 ceph-mon[128841]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2023-07-20T02:38:12.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:11 smithi017 ceph-mon[127345]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:38:12.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:11 smithi146 ceph-mon[128841]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:38:12.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:11 smithi151 ceph-mon[129163]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:38:14.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:13 smithi017 ceph-mon[127345]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T02:38:14.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:13 smithi146 ceph-mon[128841]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T02:38:14.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:13 smithi151 ceph-mon[129163]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T02:38:15.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:14 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T02:38:15.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:14 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:15.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:14 smithi017 ceph-mon[127345]: Deploying daemon osd.9 on smithi017 2023-07-20T02:38:15.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:14 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T02:38:15.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:14 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:15.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:14 smithi146 ceph-mon[128841]: Deploying daemon osd.9 on smithi017 2023-07-20T02:38:15.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:14 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T02:38:15.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:14 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:15.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:14 smithi151 ceph-mon[129163]: Deploying daemon osd.9 on smithi017 2023-07-20T02:38:16.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:15 smithi017 ceph-mon[127345]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:38:16.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:15 smithi146 ceph-mon[128841]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:38:16.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:15 smithi151 ceph-mon[129163]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:38:18.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:17 smithi017 ceph-mon[127345]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:38:18.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:17 smithi146 ceph-mon[128841]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:38:18.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:17 smithi151 ceph-mon[129163]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:38:19.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:19.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:19.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:19.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:19.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:19.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:19.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:19.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:19.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:20.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:20 smithi017 ceph-mon[127345]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-20T02:38:20.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:20 smithi146 ceph-mon[128841]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-20T02:38:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:20 smithi151 ceph-mon[129163]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-20T02:38:21.115 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.116+0000 7ff1f6ffd700 1 -- 172.21.15.17:0/3050680132 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff200060bb0 con 0x7ff1ec0a11d0 2023-07-20T02:38:21.115 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 9 on host 'smithi017' 2023-07-20T02:38:21.118 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff1ec0a11d0 msgr2=0x7ff1ec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:21.118 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff1ec0a11d0 0x7ff1ec0a3690 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff200069310 tx=0x7ff1fc009f10 comp rx=0 tx=0).stop 2023-07-20T02:38:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2000ff700 msgr2=0x7ff200068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2000ff700 0x7ff200068d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000d740 tx=0x7ff1f000da50 comp rx=0 tx=0).stop 2023-07-20T02:38:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 shutdown_connections 2023-07-20T02:38:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff1ec0a11d0 0x7ff1ec0a3690 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2001012a0 0x7ff20019d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:21.119 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2000ff700 0x7ff200068d90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 --2- 172.21.15.17:0/3050680132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff200100900 0x7ff20019b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 >> 172.21.15.17:0/3050680132 conn(0x7ff2000fac80 msgr2=0x7ff200103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:38:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 shutdown_connections 2023-07-20T02:38:21.120 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:38:21.119+0000 7ff207d16700 1 -- 172.21.15.17:0/3050680132 wait complete. 2023-07-20T02:38:21.553 DEBUG:teuthology.orchestra.run.smithi017:osd.9> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.9.service 2023-07-20T02:38:21.555 INFO:tasks.cephadm:Deploying osd.10 on smithi146 with /dev/vg_nvme/lv_1... 2023-07-20T02:38:21.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T02:38:21.588 INFO:journalctl@ceph.osd.9.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:38:22.119 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:38:22.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:22 smithi151 ceph-mon[129163]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:22.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:22 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:22 smithi146 ceph-mon[128841]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:22.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:22 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:22 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:22 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:22 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.453 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:38:22 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:38:22.328+0000 7fe8ea084640 -1 osd.9 0 log_to_monitors true 2023-07-20T02:38:22.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:22 smithi017 ceph-mon[127345]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:22.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:22 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:22 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:22 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:22 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:23 smithi146 ceph-mon[128841]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:38:23.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:23 smithi151 ceph-mon[129163]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:38:23.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:23 smithi017 ceph-mon[127345]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:38:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T02:38:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T02:38:23.786 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2023-07-20T02:38:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2023-07-20T02:38:23.787 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111389 s, 941 MB/s 2023-07-20T02:38:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2023-07-20T02:38:24.203 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:38:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:38:24.107+0000 7fe8deefb700 -1 osd.9 0 waiting for initial osdmap 2023-07-20T02:38:24.204 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:38:24 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:38:24.121+0000 7fe8d94ee700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:38:24.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:24 smithi017 ceph-mon[127345]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:24.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:24 smithi017 ceph-mon[127345]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T02:38:24.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:24 smithi017 ceph-mon[127345]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T02:38:24.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:24 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:24.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:24 smithi017 ceph-mon[127345]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:38:24.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi146:vg_nvme/lv_1 2023-07-20T02:38:24.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:24 smithi151 ceph-mon[129163]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:24.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:24 smithi151 ceph-mon[129163]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T02:38:24.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:24 smithi151 ceph-mon[129163]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T02:38:24.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:24 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:24.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:24 smithi151 ceph-mon[129163]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:38:24.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:24 smithi146 ceph-mon[128841]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:24.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:24 smithi146 ceph-mon[128841]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T02:38:24.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:24 smithi146 ceph-mon[128841]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T02:38:24.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:24 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:24.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:24 smithi146 ceph-mon[128841]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2023-07-20T02:38:24.834 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:38:25.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:38:25.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T02:38:25.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:25.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:25.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi017 2023-07-20T02:38:25.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi017 to 4293M 2023-07-20T02:38:25.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:25.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:25.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:25.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:25.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:25.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:38:25.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T02:38:25.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:25.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi017 2023-07-20T02:38:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi017 to 4293M 2023-07-20T02:38:25.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:25.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:25.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:25.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:25.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:25.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2023-07-20T02:38:25.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T02:38:25.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:25.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:25.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi017 2023-07-20T02:38:25.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:25.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi017 to 4293M 2023-07-20T02:38:25.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:25.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:25.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:25.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:25.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:25.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:25.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/2824881734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 msgr2=0x7f5f88108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/2824881734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f88108c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000b0d0 tx=0x7f5f8000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:38:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/2824881734 shutdown_connections 2023-07-20T02:38:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/2824881734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f88108c80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/2824881734 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f88100830 0x7f5f88100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/2824881734 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f88101a30 0x7f5f88101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/2824881734 >> 172.21.15.146:0/2824881734 conn(0x7f5f880fbdf0 msgr2=0x7f5f880fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:38:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/2824881734 shutdown_connections 2023-07-20T02:38:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.195+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/2824881734 wait complete. 2023-07-20T02:38:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 Processor -- start 2023-07-20T02:38:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 -- start start 2023-07-20T02:38:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f88100830 0x7f5f88196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f88101a30 0x7f5f88196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f8819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8819e110 con 0x7f5f88101a30 2023-07-20T02:38:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8819e250 con 0x7f5f881023d0 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8f0dc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8819e3e0 con 0x7f5f88100830 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8d679700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f8819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8d679700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f8819db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:52474/0 (socket says 172.21.15.146:52474) 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8d679700 1 -- 172.21.15.146:0/3157392822 learned_addr learned my addr 172.21.15.146:0/3157392822 (peer_addr_for_me v2:172.21.15.146:0/0) 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f8ce78700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f88100830 0x7f5f88196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.196+0000 7f5f87fff700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f88101a30 0x7f5f88196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8d679700 1 -- 172.21.15.146:0/3157392822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f88100830 msgr2=0x7f5f88196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8d679700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f88100830 0x7f5f88196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:26.200 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8d679700 1 -- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f88101a30 msgr2=0x7f5f88196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:26.200 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8d679700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f88101a30 0x7f5f88196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:26.200 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8d679700 1 -- 172.21.15.146:0/3157392822 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f80009d00 con 0x7f5f881023d0 2023-07-20T02:38:26.200 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f87fff700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f88101a30 0x7f5f88196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:38:26.200 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8d679700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f8819db40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007a00 tx=0x7f5f800094c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:38:26.200 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f85ffb700 1 -- 172.21.15.146:0/3157392822 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8000e040 con 0x7f5f881023d0 2023-07-20T02:38:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8819e660 con 0x7f5f881023d0 2023-07-20T02:38:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f85ffb700 1 -- 172.21.15.146:0/3157392822 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f80003e10 con 0x7f5f881023d0 2023-07-20T02:38:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f85ffb700 1 -- 172.21.15.146:0/3157392822 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80012ae0 con 0x7f5f881023d0 2023-07-20T02:38:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.197+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8819ead0 con 0x7f5f881023d0 2023-07-20T02:38:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.198+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f88107d80 con 0x7f5f881023d0 2023-07-20T02:38:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.199+0000 7f5f85ffb700 1 -- 172.21.15.146:0/3157392822 <== mon.2 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5f80012c40 con 0x7f5f881023d0 2023-07-20T02:38:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.200+0000 7f5f85ffb700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f780a1150 0x7f5f780a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.200+0000 7f5f85ffb700 1 -- 172.21.15.146:0/3157392822 <== mon.2 v2:172.21.15.146:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f5f800cdf50 con 0x7f5f881023d0 2023-07-20T02:38:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.200+0000 7f5f8ce78700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f780a1150 0x7f5f780a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:26.203 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.201+0000 7f5f8ce78700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f780a1150 0x7f5f780a3610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f88101890 tx=0x7f5f7c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:38:26.203 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.203+0000 7f5f85ffb700 1 -- 172.21.15.146:0/3157392822 <== mon.2 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5f80097f30 con 0x7f5f881023d0 2023-07-20T02:38:26.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:26 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:38:26.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:26 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:38:26.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:26 smithi146 ceph-mon[128841]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:26.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:26 smithi146 ceph-mon[128841]: osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476] boot 2023-07-20T02:38:26.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:26 smithi146 ceph-mon[128841]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T02:38:26.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:26 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:26.396 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:26.396+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f88060bb0 con 0x7f5f780a1150 2023-07-20T02:38:26.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:26 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:38:26.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:26 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:38:26.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:26 smithi017 ceph-mon[127345]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:26.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:26 smithi017 ceph-mon[127345]: osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476] boot 2023-07-20T02:38:26.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:26 smithi017 ceph-mon[127345]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T02:38:26.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:26 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:26.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:26 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:38:26.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:26 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:38:26.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:26 smithi151 ceph-mon[129163]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:38:26.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:26 smithi151 ceph-mon[129163]: osd.9 [v2:172.21.15.17:6826/1403136476,v1:172.21.15.17:6827/1403136476] boot 2023-07-20T02:38:26.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:26 smithi151 ceph-mon[129163]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T02:38:26.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:26 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:38:27.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:27 smithi146 ceph-mon[128841]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T02:38:27.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:27 smithi146 ceph-mon[128841]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:27.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:27 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:27.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:27 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:27.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:27 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:27 smithi017 ceph-mon[127345]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T02:38:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:27 smithi017 ceph-mon[127345]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:27 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:27.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:27 smithi151 ceph-mon[129163]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T02:38:27.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:27 smithi151 ceph-mon[129163]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:27.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:27.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:27.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:27 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:28.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:28 smithi017 ceph-mon[127345]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:28.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:28 smithi146 ceph-mon[128841]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:28.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:28 smithi151 ceph-mon[129163]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:29.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:29 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/3113487050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]: dispatch 2023-07-20T02:38:29.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:29 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]: dispatch 2023-07-20T02:38:29.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:29 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]': finished 2023-07-20T02:38:29.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:29 smithi017 ceph-mon[127345]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T02:38:29.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:29 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:29.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:29 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/3113487050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]: dispatch 2023-07-20T02:38:29.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:29 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]: dispatch 2023-07-20T02:38:29.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:29 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]': finished 2023-07-20T02:38:29.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:29 smithi146 ceph-mon[128841]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T02:38:29.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:29 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:29.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:29 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/3113487050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]: dispatch 2023-07-20T02:38:29.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:29 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]: dispatch 2023-07-20T02:38:29.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:29 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ee41a91-3ac3-4262-bff2-7531048859c6"}]': finished 2023-07-20T02:38:29.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:29 smithi151 ceph-mon[129163]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T02:38:29.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:29 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:30.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:30 smithi017 ceph-mon[127345]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:30.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:30 smithi017 ceph-mon[127345]: from='client.? 172.21.15.146:0/49854212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:30.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:30 smithi146 ceph-mon[128841]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:30.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:30 smithi146 ceph-mon[128841]: from='client.? 172.21.15.146:0/49854212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:30.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:30 smithi151 ceph-mon[129163]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:30.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:30 smithi151 ceph-mon[129163]: from='client.? 172.21.15.146:0/49854212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:32.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:32 smithi017 ceph-mon[127345]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:32.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:32 smithi146 ceph-mon[128841]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:32.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:32 smithi151 ceph-mon[129163]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:34.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:34 smithi146 ceph-mon[128841]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:34.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:34 smithi017 ceph-mon[127345]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:34.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:34 smithi151 ceph-mon[129163]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:35.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:35 smithi146 ceph-mon[128841]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:36.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:35 smithi151 ceph-mon[129163]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:36.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:35 smithi017 ceph-mon[127345]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:36.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T02:38:36.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:36 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:36.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:36 smithi146 ceph-mon[128841]: Deploying daemon osd.10 on smithi146 2023-07-20T02:38:37.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T02:38:37.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:36 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:37.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:36 smithi151 ceph-mon[129163]: Deploying daemon osd.10 on smithi146 2023-07-20T02:38:37.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T02:38:37.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:36 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:37.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:36 smithi017 ceph-mon[127345]: Deploying daemon osd.10 on smithi146 2023-07-20T02:38:38.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:37 smithi151 ceph-mon[129163]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:38.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:37 smithi017 ceph-mon[127345]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:38.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:37 smithi146 ceph-mon[128841]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:40.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:39 smithi146 ceph-mon[128841]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:40.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:39 smithi017 ceph-mon[127345]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:40.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:39 smithi151 ceph-mon[129163]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:42.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:42.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:42.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:41 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:42.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:42.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:42.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:41 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:42.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:41 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:43.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:42 smithi146 ceph-mon[128841]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:43.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:42 smithi151 ceph-mon[129163]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:43.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:42 smithi017 ceph-mon[127345]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:43.811 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.811+0000 7f5f85ffb700 1 -- 172.21.15.146:0/3157392822 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f5f88060bb0 con 0x7f5f780a1150 2023-07-20T02:38:43.811 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 10 on host 'smithi146' 2023-07-20T02:38:43.813 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f780a1150 msgr2=0x7f5f780a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:43.814 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f780a1150 0x7f5f780a3610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f88101890 tx=0x7f5f7c00b410 comp rx=0 tx=0).stop 2023-07-20T02:38:43.814 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 msgr2=0x7f5f8819db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:43.814 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f8819db40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007a00 tx=0x7f5f800094c0 comp rx=0 tx=0).stop 2023-07-20T02:38:43.814 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 shutdown_connections 2023-07-20T02:38:43.814 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f780a1150 0x7f5f780a3610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:43.814 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f881023d0 0x7f5f8819db40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:43.814 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5f88100830 0x7f5f88196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:43.815 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 --2- 172.21.15.146:0/3157392822 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5f88101a30 0x7f5f88196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:43.815 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 >> 172.21.15.146:0/3157392822 conn(0x7f5f880fbdf0 msgr2=0x7f5f88104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:38:43.815 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 shutdown_connections 2023-07-20T02:38:43.815 INFO:teuthology.orchestra.run.smithi146.stderr:2023-07-20T02:38:43.814+0000 7f5f8f0dc700 1 -- 172.21.15.146:0/3157392822 wait complete. 2023-07-20T02:38:44.325 DEBUG:teuthology.orchestra.run.smithi146:osd.10> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.10.service 2023-07-20T02:38:44.327 INFO:tasks.cephadm:Deploying osd.11 on smithi151 with /dev/vg_nvme/lv_1... 2023-07-20T02:38:44.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T02:38:44.358 INFO:journalctl@ceph.osd.10.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:38:44.894 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:38:44.948 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:38:44 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:38:44.739+0000 7f8498517640 -1 osd.10 0 log_to_monitors true 2023-07-20T02:38:44.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:44 smithi146 ceph-mon[128841]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:44.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:44.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:44.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:44.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:44.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:44 smithi017 ceph-mon[127345]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:44.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:44.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:44.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:44.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:45.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:44 smithi151 ceph-mon[129163]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:45.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:45.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:45.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:45.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:45.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:45 smithi151 ceph-mon[129163]: from='osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:38:45.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:45 smithi151 ceph-mon[129163]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:38:45.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:45 smithi017 ceph-mon[127345]: from='osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:38:45.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:45 smithi017 ceph-mon[127345]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:38:45.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:45 smithi146 ceph-mon[128841]: from='osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:38:45.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:45 smithi146 ceph-mon[128841]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:38:46.542 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T02:38:46.542 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T02:38:46.543 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2023-07-20T02:38:46.543 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2023-07-20T02:38:46.543 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111015 s, 945 MB/s 2023-07-20T02:38:46.543 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2023-07-20T02:38:46.818 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:38:46.631+0000 7f848d38e700 -1 osd.10 0 waiting for initial osdmap 2023-07-20T02:38:46.818 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:38:46.645+0000 7f8488182700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:38:46.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-mon[128841]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:46.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-mon[128841]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T02:38:46.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-mon[128841]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T02:38:46.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:46.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-mon[128841]: from='osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:38:46.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:46 smithi146 ceph-mon[128841]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:38:46.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:46 smithi151 ceph-mon[129163]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:46.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:46 smithi151 ceph-mon[129163]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T02:38:46.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:46 smithi151 ceph-mon[129163]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T02:38:46.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:46 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:46.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:46 smithi151 ceph-mon[129163]: from='osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:38:46.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:46 smithi151 ceph-mon[129163]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:38:46.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi151:vg_nvme/lv_1 2023-07-20T02:38:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:46 smithi017 ceph-mon[127345]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:46 smithi017 ceph-mon[127345]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T02:38:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:46 smithi017 ceph-mon[127345]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T02:38:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:46 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:46 smithi017 ceph-mon[127345]: from='osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:38:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:46 smithi017 ceph-mon[127345]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2023-07-20T02:38:47.506 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:38:47.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:38:47.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T02:38:47.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:47.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:47.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:47.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:47.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:47.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:47.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:47.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:47.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:47.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:38:48.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T02:38:48.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:48.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2023-07-20T02:38:48.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T02:38:48.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:48.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:48.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:38:48.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:48.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:48.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:38:48.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:48.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:38:48.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:38:48.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 -- 172.21.15.151:0/1464835026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 msgr2=0x7fa154107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:48.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 --2- 172.21.15.151:0/1464835026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa154107b90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00b0a0 tx=0x7fa14c00b3b0 comp rx=0 tx=0).stop 2023-07-20T02:38:48.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 -- 172.21.15.151:0/1464835026 shutdown_connections 2023-07-20T02:38:48.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 --2- 172.21.15.151:0/1464835026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa154107b90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:48.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 --2- 172.21.15.151:0/1464835026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa154100cd0 0x7fa1541010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:48.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 --2- 172.21.15.151:0/1464835026 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa1540ff930 0x7fa1540ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:48.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 -- 172.21.15.151:0/1464835026 >> 172.21.15.151:0/1464835026 conn(0x7fa1540fa420 msgr2=0x7fa1540fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:38:48.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 -- 172.21.15.151:0/1464835026 shutdown_connections 2023-07-20T02:38:48.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.710+0000 7fa15aad7700 1 -- 172.21.15.151:0/1464835026 wait complete. 2023-07-20T02:38:48.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 Processor -- start 2023-07-20T02:38:48.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 -- start start 2023-07-20T02:38:48.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa1540ff930 0x7fa1540ff380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:48.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa15419b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:48.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa154100cd0 0x7fa15419d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:48.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15419dfa0 con 0x7fa1540ff930 2023-07-20T02:38:48.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15419e110 con 0x7fa154100cd0 2023-07-20T02:38:48.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa15aad7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15419e2d0 con 0x7fa1541002f0 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa153fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa15419b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa153fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa15419b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41096/0 (socket says 172.21.15.151:41096) 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa153fff700 1 -- 172.21.15.151:0/488008476 learned_addr learned my addr 172.21.15.151:0/488008476 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa158873700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa1540ff930 0x7fa1540ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.711+0000 7fa159074700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa154100cd0 0x7fa15419d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa153fff700 1 -- 172.21.15.151:0/488008476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa154100cd0 msgr2=0x7fa15419d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa153fff700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa154100cd0 0x7fa15419d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:48.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa153fff700 1 -- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa1540ff930 msgr2=0x7fa1540ff380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:38:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa153fff700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa1540ff930 0x7fa1540ff380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:38:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa153fff700 1 -- 172.21.15.151:0/488008476 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c009d00 con 0x7fa1541002f0 2023-07-20T02:38:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa153fff700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa15419b1c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa14000b500 tx=0x7fa14000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:38:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa151ffb700 1 -- 172.21.15.151:0/488008476 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140011830 con 0x7fa1541002f0 2023-07-20T02:38:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15419e5b0 con 0x7fa1541002f0 2023-07-20T02:38:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa151ffb700 1 -- 172.21.15.151:0/488008476 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa140011e50 con 0x7fa1541002f0 2023-07-20T02:38:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa151ffb700 1 -- 172.21.15.151:0/488008476 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140010dc0 con 0x7fa1541002f0 2023-07-20T02:38:48.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.712+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15419eb30 con 0x7fa1541002f0 2023-07-20T02:38:48.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.713+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15410af00 con 0x7fa1541002f0 2023-07-20T02:38:48.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.714+0000 7fa151ffb700 1 -- 172.21.15.151:0/488008476 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa140010380 con 0x7fa1541002f0 2023-07-20T02:38:48.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.715+0000 7fa151ffb700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa1440a1150 0x7fa1440a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:38:48.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.715+0000 7fa151ffb700 1 -- 172.21.15.151:0/488008476 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fa1400cbcd0 con 0x7fa1541002f0 2023-07-20T02:38:48.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.715+0000 7fa158873700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa1440a1150 0x7fa1440a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:38:48.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.715+0000 7fa158873700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa1440a1150 0x7fa1440a3610 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa154067d00 tx=0x7fa14800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:38:48.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.719+0000 7fa151ffb700 1 -- 172.21.15.151:0/488008476 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa140095a50 con 0x7fa1541002f0 2023-07-20T02:38:48.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:38:48.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:38:48.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:48.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi146 2023-07-20T02:38:48.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi146 to 4292M 2023-07-20T02:38:48.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:48.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638] boot 2023-07-20T02:38:48.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T02:38:48.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:48 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:48.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:38:48.914+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa154060bb0 con 0x7fa1440a1150 2023-07-20T02:38:48.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:38:48.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:38:48.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:48.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi146 2023-07-20T02:38:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi146 to 4292M 2023-07-20T02:38:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638] boot 2023-07-20T02:38:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T02:38:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:48 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:49.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:38:49.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:38:49.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:38:49.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi146 2023-07-20T02:38:49.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi146 to 4292M 2023-07-20T02:38:49.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:49.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: osd.10 [v2:172.21.15.146:6824/2995236638,v1:172.21.15.146:6825/2995236638] boot 2023-07-20T02:38:49.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T02:38:49.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:48 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:38:49.902 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:49 smithi151 ceph-mon[129163]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T02:38:49.902 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:49.903 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:49.903 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:49.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:49 smithi017 ceph-mon[127345]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T02:38:49.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:49.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:49.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:50.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:49 smithi146 ceph-mon[128841]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T02:38:50.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:38:50.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:38:50.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:50.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:50 smithi151 ceph-mon[129163]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:38:50.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:50 smithi151 ceph-mon[129163]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:50.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:50 smithi151 ceph-mon[129163]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T02:38:50.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:50 smithi151 ceph-mon[129163]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T02:38:50.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:50 smithi017 ceph-mon[127345]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:38:50.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:50 smithi017 ceph-mon[127345]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:50.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:50 smithi017 ceph-mon[127345]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T02:38:50.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:50 smithi017 ceph-mon[127345]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T02:38:51.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:50 smithi146 ceph-mon[128841]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:38:51.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:50 smithi146 ceph-mon[128841]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:38:51.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:50 smithi146 ceph-mon[128841]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T02:38:51.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:50 smithi146 ceph-mon[128841]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T02:38:52.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:51 smithi017 ceph-mon[127345]: osdmap e74: 11 total, 11 up, 11 in 2023-07-20T02:38:52.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:51 smithi017 ceph-mon[127345]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:38:52.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:51 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/2186540567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]: dispatch 2023-07-20T02:38:52.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:51 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]: dispatch 2023-07-20T02:38:52.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:51 smithi017 ceph-mon[127345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]': finished 2023-07-20T02:38:52.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:51 smithi017 ceph-mon[127345]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T02:38:52.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:51 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:38:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:51 smithi146 ceph-mon[128841]: osdmap e74: 11 total, 11 up, 11 in 2023-07-20T02:38:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:51 smithi146 ceph-mon[128841]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:38:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:51 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/2186540567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]: dispatch 2023-07-20T02:38:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:51 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]: dispatch 2023-07-20T02:38:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:51 smithi146 ceph-mon[128841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]': finished 2023-07-20T02:38:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:51 smithi146 ceph-mon[128841]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T02:38:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:51 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:38:52.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:51 smithi151 ceph-mon[129163]: osdmap e74: 11 total, 11 up, 11 in 2023-07-20T02:38:52.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:51 smithi151 ceph-mon[129163]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:38:52.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:51 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/2186540567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]: dispatch 2023-07-20T02:38:52.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:51 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]: dispatch 2023-07-20T02:38:52.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:51 smithi151 ceph-mon[129163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "282d07a4-1bb0-4239-9c1e-adeebe9d7432"}]': finished 2023-07-20T02:38:52.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:51 smithi151 ceph-mon[129163]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T02:38:52.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:51 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:38:53.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:52 smithi017 ceph-mon[127345]: from='client.? 172.21.15.151:0/2892947864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:53.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:52 smithi146 ceph-mon[128841]: from='client.? 172.21.15.151:0/2892947864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:53.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:52 smithi151 ceph-mon[129163]: from='client.? 172.21.15.151:0/2892947864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:38:54.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:53 smithi017 ceph-mon[127345]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:38:54.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:53 smithi146 ceph-mon[128841]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:38:54.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:53 smithi151 ceph-mon[129163]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:38:56.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:55 smithi151 ceph-mon[129163]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T02:38:56.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:55 smithi017 ceph-mon[127345]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T02:38:56.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:55 smithi146 ceph-mon[128841]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T02:38:58.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:57 smithi151 ceph-mon[129163]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T02:38:58.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:57 smithi017 ceph-mon[127345]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T02:38:58.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:57 smithi146 ceph-mon[128841]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T02:38:58.999 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T02:38:59.000 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:58 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:59.000 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:58 smithi151 ceph-mon[129163]: Deploying daemon osd.11 on smithi151 2023-07-20T02:38:59.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T02:38:59.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:58 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:59.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:58 smithi017 ceph-mon[127345]: Deploying daemon osd.11 on smithi151 2023-07-20T02:38:59.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T02:38:59.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:58 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:38:59.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:58 smithi146 ceph-mon[128841]: Deploying daemon osd.11 on smithi151 2023-07-20T02:39:00.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:38:59 smithi017 ceph-mon[127345]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:39:00.270 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:38:59 smithi151 ceph-mon[129163]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:39:00.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:38:59 smithi146 ceph-mon[128841]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:39:02.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:01 smithi151 ceph-mon[129163]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:39:02.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:01 smithi017 ceph-mon[127345]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:39:02.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:01 smithi146 ceph-mon[128841]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:39:03.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:03.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:03.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:03 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:04.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:04.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:04.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:03 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:04.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:04.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:04.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:03 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:05.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:04 smithi146 ceph-mon[128841]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T02:39:05.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:04 smithi151 ceph-mon[129163]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T02:39:05.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:04 smithi017 ceph-mon[127345]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T02:39:05.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.580+0000 7fa151ffb700 1 -- 172.21.15.151:0/488008476 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fa154060bb0 con 0x7fa1440a1150 2023-07-20T02:39:05.582 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 11 on host 'smithi151' 2023-07-20T02:39:05.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa1440a1150 msgr2=0x7fa1440a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:05.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa1440a1150 0x7fa1440a3610 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa154067d00 tx=0x7fa14800a380 comp rx=0 tx=0).stop 2023-07-20T02:39:05.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 msgr2=0x7fa15419b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:05.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa15419b1c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa14000b500 tx=0x7fa14000b810 comp rx=0 tx=0).stop 2023-07-20T02:39:05.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 shutdown_connections 2023-07-20T02:39:05.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fa1440a1150 0x7fa1440a3610 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:05.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa154100cd0 0x7fa15419d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:05.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa1540ff930 0x7fa1540ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:05.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 --2- 172.21.15.151:0/488008476 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1541002f0 0x7fa15419b1c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:05.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 >> 172.21.15.151:0/488008476 conn(0x7fa1540fa420 msgr2=0x7fa1540fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:05.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 shutdown_connections 2023-07-20T02:39:05.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:05.583+0000 7fa15aad7700 1 -- 172.21.15.151:0/488008476 wait complete. 2023-07-20T02:39:06.010 DEBUG:teuthology.orchestra.run.smithi151:osd.11> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.11.service 2023-07-20T02:39:06.012 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-20T02:39:06.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T02:39:06.044 INFO:journalctl@ceph.osd.11.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:39:06.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:06.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:06 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:06.529 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:06.530 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.530 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.530 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:06.531 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:06.531 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.531 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.531 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.532 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:06.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:06.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:06.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:06.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:06.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:06 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:06.590 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:39:06.823 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:39:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:39:06.676+0000 7f61d1fab640 -1 osd.11 0 log_to_monitors true 2023-07-20T02:39:07.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:07 smithi151 ceph-mon[129163]: from='osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:39:07.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:07 smithi151 ceph-mon[129163]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:39:07.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:07 smithi017 ceph-mon[127345]: from='osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:39:07.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:07 smithi017 ceph-mon[127345]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:39:07.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:07 smithi146 ceph-mon[128841]: from='osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:39:07.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:07 smithi146 ceph-mon[128841]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:39:07.676 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 -- 172.21.15.17:0/51369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c106530 msgr2=0x7ffa7c108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:07.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 --2- 172.21.15.17:0/51369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c106530 0x7ffa7c108960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400b0a0 tx=0x7ffa7400b3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 -- 172.21.15.17:0/51369942 shutdown_connections 2023-07-20T02:39:07.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 --2- 172.21.15.17:0/51369942 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c106530 0x7ffa7c108960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 --2- 172.21.15.17:0/51369942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa7c1012d0 0x7ffa7c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 --2- 172.21.15.17:0/51369942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa7c103c00 0x7ffa7c105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.677 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 -- 172.21.15.17:0/51369942 >> 172.21.15.17:0/51369942 conn(0x7ffa7c0fac80 msgr2=0x7ffa7c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:07.678 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 -- 172.21.15.17:0/51369942 shutdown_connections 2023-07-20T02:39:07.678 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.675+0000 7ffa833d3700 1 -- 172.21.15.17:0/51369942 wait complete. 2023-07-20T02:39:07.679 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 Processor -- start 2023-07-20T02:39:07.679 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 -- start start 2023-07-20T02:39:07.679 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c1012d0 0x7ffa7c196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa7c103c00 0x7ffa7c196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa7c106530 0x7ffa7c19dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c19e1b0 con 0x7ffa7c1012d0 2023-07-20T02:39:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c19e2f0 con 0x7ffa7c106530 2023-07-20T02:39:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa833d3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c19e480 con 0x7ffa7c103c00 2023-07-20T02:39:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c1012d0 0x7ffa7c196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:07.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c1012d0 0x7ffa7c196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43216/0 (socket says 172.21.15.17:43216) 2023-07-20T02:39:07.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 -- 172.21.15.17:0/3861226196 learned_addr learned my addr 172.21.15.17:0/3861226196 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:07.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8096e700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa7c103c00 0x7ffa7c196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:07.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa81970700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa7c106530 0x7ffa7c19dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:07.681 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 -- 172.21.15.17:0/3861226196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa7c103c00 msgr2=0x7ffa7c196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:07.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa7c103c00 0x7ffa7c196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 -- 172.21.15.17:0/3861226196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa7c106530 msgr2=0x7ffa7c19dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:07.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa7c106530 0x7ffa7c19dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 -- 172.21.15.17:0/3861226196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74009d00 con 0x7ffa7c1012d0 2023-07-20T02:39:07.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.676+0000 7ffa8116f700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c1012d0 0x7ffa7c196620 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800b560 tx=0x7ffa7800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:07.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.677+0000 7ffa6e7fc700 1 -- 172.21.15.17:0/3861226196 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa78011830 con 0x7ffa7c1012d0 2023-07-20T02:39:07.682 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.677+0000 7ffa6e7fc700 1 -- 172.21.15.17:0/3861226196 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa78011e50 con 0x7ffa7c1012d0 2023-07-20T02:39:07.683 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.677+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7c19e760 con 0x7ffa7c1012d0 2023-07-20T02:39:07.683 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.677+0000 7ffa6e7fc700 1 -- 172.21.15.17:0/3861226196 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7800f2e0 con 0x7ffa7c1012d0 2023-07-20T02:39:07.683 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.677+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7c19ec60 con 0x7ffa7c1012d0 2023-07-20T02:39:07.683 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.678+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7c0fc800 con 0x7ffa7c1012d0 2023-07-20T02:39:07.684 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.679+0000 7ffa6e7fc700 1 -- 172.21.15.17:0/3861226196 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ffa78010580 con 0x7ffa7c1012d0 2023-07-20T02:39:07.684 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.679+0000 7ffa6e7fc700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ffa680a1150 0x7ffa680a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:07.684 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.679+0000 7ffa8096e700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ffa680a1150 0x7ffa680a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:07.685 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.679+0000 7ffa6e7fc700 1 -- 172.21.15.17:0/3861226196 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6831+0+0 (secure 0 0 0) 0x7ffa780cc330 con 0x7ffa7c1012d0 2023-07-20T02:39:07.685 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.680+0000 7ffa8096e700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ffa680a1150 0x7ffa680a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffa70007c60 tx=0x7ffa700073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:07.685 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.683+0000 7ffa6e7fc700 1 -- 172.21.15.17:0/3861226196 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffa78096050 con 0x7ffa7c1012d0 2023-07-20T02:39:07.883 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.882+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ffa7c02c970 con 0x7ffa7c1012d0 2023-07-20T02:39:07.884 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.882+0000 7ffa6e7fc700 1 -- 172.21.15.17:0/3861226196 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7ffa78099670 con 0x7ffa7c1012d0 2023-07-20T02:39:07.884 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ffa680a1150 msgr2=0x7ffa680a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ffa680a1150 0x7ffa680a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffa70007c60 tx=0x7ffa700073d0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c1012d0 msgr2=0x7ffa7c196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c1012d0 0x7ffa7c196620 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800b560 tx=0x7ffa7800b870 comp rx=0 tx=0).stop 2023-07-20T02:39:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 shutdown_connections 2023-07-20T02:39:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ffa680a1150 0x7ffa680a3610 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa7c106530 0x7ffa7c19dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ffa7c1012d0 0x7ffa7c196620 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 --2- 172.21.15.17:0/3861226196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa7c103c00 0x7ffa7c196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 >> 172.21.15.17:0/3861226196 conn(0x7ffa7c0fac80 msgr2=0x7ffa7c101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 shutdown_connections 2023-07-20T02:39:07.889 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:07.885+0000 7ffa833d3700 1 -- 172.21.15.17:0/3861226196 wait complete. 2023-07-20T02:39:08.255 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":76,"num_osds":12,"num_up_osds":11,"osd_up_since":1689820727,"num_in_osds":12,"osd_in_since":1689820731,"num_remapped_pgs":0} 2023-07-20T02:39:08.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:08 smithi146 ceph-mon[128841]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:08.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:08 smithi146 ceph-mon[128841]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T02:39:08.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:08 smithi146 ceph-mon[128841]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T02:39:08.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:08 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:08.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:08 smithi146 ceph-mon[128841]: from='osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:39:08.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:08 smithi146 ceph-mon[128841]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:39:08.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:08 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3861226196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:39:08.612 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:39:08.259+0000 7f61c6e22700 -1 osd.11 0 waiting for initial osdmap 2023-07-20T02:39:08.612 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:39:08.275+0000 7f61c1c16700 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:39:08.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-mon[129163]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:08.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-mon[129163]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T02:39:08.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-mon[129163]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T02:39:08.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:08.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-mon[129163]: from='osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:39:08.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-mon[129163]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:39:08.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:08 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3861226196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:39:08.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:08 smithi017 ceph-mon[127345]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:08.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:08 smithi017 ceph-mon[127345]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T02:39:08.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:08 smithi017 ceph-mon[127345]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T02:39:08.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:08 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:08.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:08 smithi017 ceph-mon[127345]: from='osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:39:08.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:08 smithi017 ceph-mon[127345]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2023-07-20T02:39:08.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:08 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3861226196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:39:09.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T02:39:09.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:39:09.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: osdmap e77: 12 total, 11 up, 12 in 2023-07-20T02:39:09.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:09.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:09.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: Detected new or changed devices on smithi151 2023-07-20T02:39:09.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: Adjusting osd_memory_target on smithi151 to 4292M 2023-07-20T02:39:09.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:09.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:09.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:09 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:39:09.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: osdmap e77: 12 total, 11 up, 12 in 2023-07-20T02:39:09.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:09.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:09.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: Detected new or changed devices on smithi151 2023-07-20T02:39:09.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: Adjusting osd_memory_target on smithi151 to 4292M 2023-07-20T02:39:09.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:09.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:09.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:09 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2023-07-20T02:39:09.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: osdmap e77: 12 total, 11 up, 12 in 2023-07-20T02:39:09.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:09.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:09.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: Detected new or changed devices on smithi151 2023-07-20T02:39:09.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:39:09.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: Adjusting osd_memory_target on smithi151 to 4292M 2023-07-20T02:39:09.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:09.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:09.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:09 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:09.830 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:39:10.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:10 smithi146 ceph-mon[128841]: purged_snaps scrub starts 2023-07-20T02:39:10.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:10 smithi146 ceph-mon[128841]: purged_snaps scrub ok 2023-07-20T02:39:10.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:10 smithi146 ceph-mon[128841]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:10.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:10 smithi146 ceph-mon[128841]: osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079] boot 2023-07-20T02:39:10.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:10 smithi146 ceph-mon[128841]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T02:39:10.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:10 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:10.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:10 smithi151 ceph-mon[129163]: purged_snaps scrub starts 2023-07-20T02:39:10.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:10 smithi151 ceph-mon[129163]: purged_snaps scrub ok 2023-07-20T02:39:10.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:10 smithi151 ceph-mon[129163]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:10.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:10 smithi151 ceph-mon[129163]: osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079] boot 2023-07-20T02:39:10.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:10 smithi151 ceph-mon[129163]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T02:39:10.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:10 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:10.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:10 smithi017 ceph-mon[127345]: purged_snaps scrub starts 2023-07-20T02:39:10.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:10 smithi017 ceph-mon[127345]: purged_snaps scrub ok 2023-07-20T02:39:10.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:10 smithi017 ceph-mon[127345]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:39:10.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:10 smithi017 ceph-mon[127345]: osd.11 [v2:172.21.15.151:6824/1663985079,v1:172.21.15.151:6825/1663985079] boot 2023-07-20T02:39:10.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:10 smithi017 ceph-mon[127345]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T02:39:10.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:10 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:39:11.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 -- 172.21.15.17:0/2638224589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 msgr2=0x7f4400100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:11.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 --2- 172.21.15.17:0/2638224589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400100d60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f43e80099c0 tx=0x7f43e8009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 -- 172.21.15.17:0/2638224589 shutdown_connections 2023-07-20T02:39:11.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 --2- 172.21.15.17:0/2638224589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44001012a0 0x7f4400107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.049 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 --2- 172.21.15.17:0/2638224589 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44000ff700 0x7f44000ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 --2- 172.21.15.17:0/2638224589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400100d60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 -- 172.21.15.17:0/2638224589 >> 172.21.15.17:0/2638224589 conn(0x7f44000fac80 msgr2=0x7f44000fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:11.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 -- 172.21.15.17:0/2638224589 shutdown_connections 2023-07-20T02:39:11.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.045+0000 7f4405087700 1 -- 172.21.15.17:0/2638224589 wait complete. 2023-07-20T02:39:11.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 Processor -- start 2023-07-20T02:39:11.050 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 -- start start 2023-07-20T02:39:11.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44000ff700 0x7f44001965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:11.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:11.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44001012a0 0x7f440019db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:11.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440019e130 con 0x7f4400100900 2023-07-20T02:39:11.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440019e270 con 0x7f44000ff700 2023-07-20T02:39:11.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f4405087700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440019e400 con 0x7f44001012a0 2023-07-20T02:39:11.051 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f43fe59c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:11.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f43fe59c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400196ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43234/0 (socket says 172.21.15.17:43234) 2023-07-20T02:39:11.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.046+0000 7f43fe59c700 1 -- 172.21.15.17:0/3375377916 learned_addr learned my addr 172.21.15.17:0/3375377916 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:11.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43fed9d700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44000ff700 0x7f44001965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:11.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43ff59e700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44001012a0 0x7f440019db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:11.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43fe59c700 1 -- 172.21.15.17:0/3375377916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44001012a0 msgr2=0x7f440019db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:11.052 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43fe59c700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44001012a0 0x7f440019db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43fe59c700 1 -- 172.21.15.17:0/3375377916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44000ff700 msgr2=0x7f44001965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43fe59c700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44000ff700 0x7f44001965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43fe59c700 1 -- 172.21.15.17:0/3375377916 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80095f0 con 0x7f4400100900 2023-07-20T02:39:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43ff59e700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44001012a0 0x7f440019db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:39:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43fe59c700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400196ae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f43e8003af0 tx=0x7f43e8003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43f7fff700 1 -- 172.21.15.17:0/3375377916 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e801d070 con 0x7f4400100900 2023-07-20T02:39:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440019e680 con 0x7f4400100900 2023-07-20T02:39:11.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43f7fff700 1 -- 172.21.15.17:0/3375377916 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43e8004640 con 0x7f4400100900 2023-07-20T02:39:11.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f43f7fff700 1 -- 172.21.15.17:0/3375377916 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e800f460 con 0x7f4400100900 2023-07-20T02:39:11.054 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.047+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440019ebf0 con 0x7f4400100900 2023-07-20T02:39:11.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.048+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44001077f0 con 0x7f4400100900 2023-07-20T02:39:11.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.049+0000 7f43f7fff700 1 -- 172.21.15.17:0/3375377916 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f43e80040b0 con 0x7f4400100900 2023-07-20T02:39:11.055 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.050+0000 7f43f7fff700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43ec0a1150 0x7f43ec0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:11.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.050+0000 7f43f7fff700 1 -- 172.21.15.17:0/3375377916 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43e80cde00 con 0x7f4400100900 2023-07-20T02:39:11.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.050+0000 7f43fed9d700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43ec0a1150 0x7f43ec0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:11.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.050+0000 7f43fed9d700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43ec0a1150 0x7f43ec0a3610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006fd0 tx=0x7f43f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:11.056 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.053+0000 7f43f7fff700 1 -- 172.21.15.17:0/3375377916 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f43e8097a80 con 0x7f4400100900 2023-07-20T02:39:11.252 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.251+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f440002cb70 con 0x7f4400100900 2023-07-20T02:39:11.253 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.251+0000 7f43f7fff700 1 -- 172.21.15.17:0/3375377916 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f43e809b0a0 con 0x7f4400100900 2023-07-20T02:39:11.253 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.253+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43ec0a1150 msgr2=0x7f43ec0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.253+0000 7f4405087700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43ec0a1150 0x7f43ec0a3610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006fd0 tx=0x7f43f0008040 comp rx=0 tx=0).stop 2023-07-20T02:39:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.253+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 msgr2=0x7f4400196ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.253+0000 7f4405087700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400196ae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f43e8003af0 tx=0x7f43e8003bd0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 shutdown_connections 2023-07-20T02:39:11.255 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f43ec0a1150 0x7f43ec0a3610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44001012a0 0x7f440019db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44000ff700 0x7f44001965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 --2- 172.21.15.17:0/3375377916 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f4400100900 0x7f4400196ae0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 >> 172.21.15.17:0/3375377916 conn(0x7f44000fac80 msgr2=0x7f44000fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 shutdown_connections 2023-07-20T02:39:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:11.254+0000 7f4405087700 1 -- 172.21.15.17:0/3375377916 wait complete. 2023-07-20T02:39:11.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:11 smithi146 ceph-mon[128841]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T02:39:11.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:11 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3375377916' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:39:11.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:11 smithi151 ceph-mon[129163]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T02:39:11.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:11 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3375377916' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:39:11.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:11 smithi017 ceph-mon[127345]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T02:39:11.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:11 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3375377916' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:39:11.687 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1689820749,"num_in_osds":12,"osd_in_since":1689820731,"num_remapped_pgs":0} 2023-07-20T02:39:11.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph osd dump --format=json 2023-07-20T02:39:12.250 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:39:12.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:12 smithi146 ceph-mon[128841]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:12.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:12 smithi151 ceph-mon[129163]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:12.637 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:12 smithi017 ceph-mon[127345]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:13.458 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 -- 172.21.15.17:0/791887421 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c0ff6b0 msgr2=0x7f355c0ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:13.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 --2- 172.21.15.17:0/791887421 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c0ff6b0 0x7f355c0ffb30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3550009990 tx=0x7f3550009ca0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 -- 172.21.15.17:0/791887421 shutdown_connections 2023-07-20T02:39:13.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 --2- 172.21.15.17:0/791887421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f355c100070 0x7f355c10c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 --2- 172.21.15.17:0/791887421 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f355c104580 0x7f355c104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 --2- 172.21.15.17:0/791887421 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c0ff6b0 0x7f355c0ffb30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.459 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 -- 172.21.15.17:0/791887421 >> 172.21.15.17:0/791887421 conn(0x7f355c076770 msgr2=0x7f355c076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:13.460 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 -- 172.21.15.17:0/791887421 shutdown_connections 2023-07-20T02:39:13.460 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.457+0000 7f3560ba2700 1 -- 172.21.15.17:0/791887421 wait complete. 2023-07-20T02:39:13.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 Processor -- start 2023-07-20T02:39:13.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 -- start start 2023-07-20T02:39:13.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f355c0ff6b0 0x7f355c10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:13.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f355c100070 0x7f355c10d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:13.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c104580 0x7f355c10d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:13.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c1a4d40 con 0x7f355c104580 2023-07-20T02:39:13.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c1a4eb0 con 0x7f355c0ff6b0 2023-07-20T02:39:13.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3560ba2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c1a5070 con 0x7f355c100070 2023-07-20T02:39:13.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f355ad9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c104580 0x7f355c10d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:13.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f355ad9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c104580 0x7f355c10d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43258/0 (socket says 172.21.15.17:43258) 2023-07-20T02:39:13.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f355ad9d700 1 -- 172.21.15.17:0/2857969379 learned_addr learned my addr 172.21.15.17:0/2857969379 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:13.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f3559d9b700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f355c100070 0x7f355c10d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:13.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.458+0000 7f355a59c700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f355c0ff6b0 0x7f355c10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:13.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f355ad9d700 1 -- 172.21.15.17:0/2857969379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f355c100070 msgr2=0x7f355c10d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:13.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f355ad9d700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f355c100070 0x7f355c10d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f355ad9d700 1 -- 172.21.15.17:0/2857969379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f355c0ff6b0 msgr2=0x7f355c10ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:13.467 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f355ad9d700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f355c0ff6b0 0x7f355c10ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f355ad9d700 1 -- 172.21.15.17:0/2857969379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500095f0 con 0x7f355c104580 2023-07-20T02:39:13.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f355ad9d700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c104580 0x7f355c10d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f355400ab50 tx=0x7f355400af10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:13.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f354b7fe700 1 -- 172.21.15.17:0/2857969379 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400f070 con 0x7f355c104580 2023-07-20T02:39:13.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f354b7fe700 1 -- 172.21.15.17:0/2857969379 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3554007a80 con 0x7f355c104580 2023-07-20T02:39:13.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f354b7fe700 1 -- 172.21.15.17:0/2857969379 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554005310 con 0x7f355c104580 2023-07-20T02:39:13.468 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c1a5350 con 0x7f355c104580 2023-07-20T02:39:13.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.459+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c1a5800 con 0x7f355c104580 2023-07-20T02:39:13.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.460+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c04e470 con 0x7f355c104580 2023-07-20T02:39:13.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.461+0000 7f354b7fe700 1 -- 172.21.15.17:0/2857969379 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3554004ad0 con 0x7f355c104580 2023-07-20T02:39:13.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.462+0000 7f354b7fe700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f35440a1150 0x7f35440a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:13.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.462+0000 7f355a59c700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f35440a1150 0x7f35440a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:13.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.462+0000 7f354b7fe700 1 -- 172.21.15.17:0/2857969379 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f35540cc050 con 0x7f355c104580 2023-07-20T02:39:13.469 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.462+0000 7f355a59c700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f35440a1150 0x7f35440a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f354c006fd0 tx=0x7f354c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:13.470 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.465+0000 7f354b7fe700 1 -- 172.21.15.17:0/2857969379 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3554095cd0 con 0x7f355c104580 2023-07-20T02:39:13.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.662+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f355c10e400 con 0x7f355c104580 2023-07-20T02:39:13.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.664+0000 7f354b7fe700 1 -- 172.21.15.17:0/2857969379 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18421 (secure 0 0 0) 0x7f35540992f0 con 0x7f355c104580 2023-07-20T02:39:13.665 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:13.665 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":79,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","created":"2023-07-20T02:32:50.175394+0000","modified":"2023-07-20T02:39:10.303454+0000","last_up_change":"2023-07-20T02:39:09.254013+0000","last_in_change":"2023-07-20T02:38:51.253472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:35:55.360132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb89ee2e-e4ff-4c83-9626-69cbfb526dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6803","nonce":795351069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6805","nonce":795351069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6809","nonce":795351069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6807","nonce":795351069}]},"public_addr":"172.21.15.17:6803/795351069","cluster_addr":"172.21.15.17:6805/795351069","heartbeat_back_addr":"172.21.15.17:6809/795351069","heartbeat_front_addr":"172.21.15.17:6807/795351069","state":["exists","up"]},{"osd":1,"uuid":"2d422389-0925-410c-bd28-50ad06d68cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6801","nonce":3123864696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6803","nonce":3123864696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6807","nonce":3123864696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6805","nonce":3123864696}]},"public_addr":"172.21.15.146:6801/3123864696","cluster_addr":"172.21.15.146:6803/3123864696","heartbeat_back_addr":"172.21.15.146:6807/3123864696","heartbeat_front_addr":"172.21.15.146:6805/3123864696","state":["exists","up"]},{"osd":2,"uuid":"a08d37e7-a3cd-4694-9441-b505bd167c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6801","nonce":1299864358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6803","nonce":1299864358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6807","nonce":1299864358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6805","nonce":1299864358}]},"public_addr":"172.21.15.151:6801/1299864358","cluster_addr":"172.21.15.151:6803/1299864358","heartbeat_back_addr":"172.21.15.151:6807/1299864358","heartbeat_front_addr":"172.21.15.151:6805/1299864358","state":["exists","up"]},{"osd":3,"uuid":"62ad61a1-05b0-4e38-a2eb-530485d806b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6811","nonce":604342850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6813","nonce":604342850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6817","nonce":604342850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6815","nonce":604342850}]},"public_addr":"172.21.15.17:6811/604342850","cluster_addr":"172.21.15.17:6813/604342850","heartbeat_back_addr":"172.21.15.17:6817/604342850","heartbeat_front_addr":"172.21.15.17:6815/604342850","state":["exists","up"]},{"osd":4,"uuid":"38ed288f-9c0d-415c-b52f-075d3b7599db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6809","nonce":4024595090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6811","nonce":4024595090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6815","nonce":4024595090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6813","nonce":4024595090}]},"public_addr":"172.21.15.146:6809/4024595090","cluster_addr":"172.21.15.146:6811/4024595090","heartbeat_back_addr":"172.21.15.146:6815/4024595090","heartbeat_front_addr":"172.21.15.146:6813/4024595090","state":["exists","up"]},{"osd":5,"uuid":"7da194cb-67fb-4c73-b84c-fa899c3635b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6809","nonce":1633219838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6811","nonce":1633219838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6815","nonce":1633219838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6813","nonce":1633219838}]},"public_addr":"172.21.15.151:6809/1633219838","cluster_addr":"172.21.15.151:6811/1633219838","heartbeat_back_addr":"172.21.15.151:6815/1633219838","heartbeat_front_addr":"172.21.15.151:6813/1633219838","state":["exists","up"]},{"osd":6,"uuid":"67ced483-d37a-46c1-97ae-288f28b091f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6819","nonce":3807389822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6821","nonce":3807389822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6825","nonce":3807389822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6823","nonce":3807389822}]},"public_addr":"172.21.15.17:6819/3807389822","cluster_addr":"172.21.15.17:6821/3807389822","heartbeat_back_addr":"172.21.15.17:6825/3807389822","heartbeat_front_addr":"172.21.15.17:6823/3807389822","state":["exists","up"]},{"osd":7,"uuid":"26199bbc-d82f-4e97-9218-0058426ef412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6817","nonce":708669673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6819","nonce":708669673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6823","nonce":708669673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6821","nonce":708669673}]},"public_addr":"172.21.15.146:6817/708669673","cluster_addr":"172.21.15.146:6819/708669673","heartbeat_back_addr":"172.21.15.146:6823/708669673","heartbeat_front_addr":"172.21.15.146:6821/708669673","state":["exists","up"]},{"osd":8,"uuid":"b1a33778-9ab5-45db-90ff-644794ee1cd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6817","nonce":2042667088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6819","nonce":2042667088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6823","nonce":2042667088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6821","nonce":2042667088}]},"public_addr":"172.21.15.151:6817/2042667088","cluster_addr":"172.21.15.151:6819/2042667088","heartbeat_back_addr":"172.21.15.151:6823/2042667088","heartbeat_front_addr":"172.21.15.151:6821/2042667088","state":["exists","up"]},{"osd":9,"uuid":"496d04b3-61a4-4405-aa52-5590aacf7129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6827","nonce":1403136476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6829","nonce":1403136476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6833","nonce":1403136476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6831","nonce":1403136476}]},"public_addr":"172.21.15.17:6827/1403136476","cluster_addr":"172.21.15.17:6829/1403136476","heartbeat_back_addr":"172.21.15.17:6833/1403136476","heartbeat_front_addr":"172.21.15.17:6831/1403136476","state":["exists","up"]},{"osd":10,"uuid":"6ee41a91-3ac3-4262-bff2-7531048859c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6825","nonce":2995236638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6827","nonce":2995236638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6831","nonce":2995236638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6829","nonce":2995236638}]},"public_addr":"172.21.15.146:6825/2995236638","cluster_addr":"172.21.15.146:6827/2995236638","heartbeat_back_addr":"172.21.15.146:6831/2995236638","heartbeat_front_addr":"172.21.15.146:6829/2995236638","state":["exists","up"]},{"osd":11,"uuid":"282d07a4-1bb0-4239-9c1e-adeebe9d7432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6825","nonce":1663985079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6827","nonce":1663985079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6831","nonce":1663985079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6829","nonce":1663985079}]},"public_addr":"172.21.15.151:6825/1663985079","cluster_addr":"172.21.15.151:6827/1663985079","heartbeat_back_addr":"172.21.15.151:6831/1663985079","heartbeat_front_addr":"172.21.15.151:6829/1663985079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:07.560935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:29.818964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:51.802583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:12.985262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:34.792736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:56.378329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:17.498840+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:40.421290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:02.192805+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:23.342606+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:45.695709+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:39:07.701215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2979320785":"2023-07-21T02:33:44.823257+0000","172.21.15.17:0/979357839":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6800/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/714275157":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1888486288":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/910909892":"2023-07-21T02:33:19.367142+0000","172.21.15.17:6800/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1536507180":"2023-07-21T02:33:44.823257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:39:13.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f35440a1150 msgr2=0x7f35440a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:13.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f35440a1150 0x7f35440a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f354c006fd0 tx=0x7f354c008040 comp rx=0 tx=0).stop 2023-07-20T02:39:13.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c104580 msgr2=0x7f355c10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:13.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c104580 0x7f355c10d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f355400ab50 tx=0x7f355400af10 comp rx=0 tx=0).stop 2023-07-20T02:39:13.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 shutdown_connections 2023-07-20T02:39:13.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f35440a1150 0x7f35440a3610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.669 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f355c104580 0x7f355c10d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f355c0ff6b0 0x7f355c10ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 --2- 172.21.15.17:0/2857969379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f355c100070 0x7f355c10d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:13.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 >> 172.21.15.17:0/2857969379 conn(0x7f355c076770 msgr2=0x7f355c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:13.670 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 shutdown_connections 2023-07-20T02:39:13.671 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:13.666+0000 7f3560ba2700 1 -- 172.21.15.17:0/2857969379 wait complete. 2023-07-20T02:39:14.109 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:35:55.360132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T02:39:14.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-20T02:39:14.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:14 smithi017 ceph-mon[127345]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:14.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:14 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2857969379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:14.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:14 smithi146 ceph-mon[128841]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:14.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:14 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2857969379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:14.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:14 smithi151 ceph-mon[129163]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:14.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:14 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2857969379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:14.669 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:39:15.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 -- 172.21.15.17:0/14756470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd940105ba0 msgr2=0x7fd94010d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:15.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 --2- 172.21.15.17:0/14756470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd940105ba0 0x7fd94010d5b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd93800b0d0 tx=0x7fd93800b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:15.886 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 -- 172.21.15.17:0/14756470 shutdown_connections 2023-07-20T02:39:15.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 --2- 172.21.15.17:0/14756470 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd940105ba0 0x7fd94010d5b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:15.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 --2- 172.21.15.17:0/14756470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd940068730 0x7fd940068b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:15.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 --2- 172.21.15.17:0/14756470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9401051e0 0x7fd940105660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:15.887 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 -- 172.21.15.17:0/14756470 >> 172.21.15.17:0/14756470 conn(0x7fd940076770 msgr2=0x7fd940076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:15.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 -- 172.21.15.17:0/14756470 shutdown_connections 2023-07-20T02:39:15.888 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.884+0000 7fd947767700 1 -- 172.21.15.17:0/14756470 wait complete. 2023-07-20T02:39:15.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.885+0000 7fd947767700 1 Processor -- start 2023-07-20T02:39:15.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.885+0000 7fd947767700 1 -- start start 2023-07-20T02:39:15.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd947767700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd940068730 0x7fd940196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:15.890 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd947767700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9401051e0 0x7fd940196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:15.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd947767700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd940105ba0 0x7fd94019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:15.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd947767700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94019b900 con 0x7fd9401051e0 2023-07-20T02:39:15.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd947767700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94019ba70 con 0x7fd940105ba0 2023-07-20T02:39:15.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd947767700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94019bc30 con 0x7fd940068730 2023-07-20T02:39:15.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9401051e0 0x7fd940196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:15.891 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9401051e0 0x7fd940196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43268/0 (socket says 172.21.15.17:43268) 2023-07-20T02:39:15.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 -- 172.21.15.17:0/95410622 learned_addr learned my addr 172.21.15.17:0/95410622 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:15.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd945d04700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd940105ba0 0x7fd94019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:15.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd945503700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd940068730 0x7fd940196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:15.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 -- 172.21.15.17:0/95410622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd940068730 msgr2=0x7fd940196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:15.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd940068730 0x7fd940196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:15.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 -- 172.21.15.17:0/95410622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd940105ba0 msgr2=0x7fd94019b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:15.892 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd940105ba0 0x7fd94019b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:15.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 -- 172.21.15.17:0/95410622 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd938009d00 con 0x7fd9401051e0 2023-07-20T02:39:15.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd944d02700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9401051e0 0x7fd940196e90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd93000b530 tx=0x7fd93000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:15.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd9367fc700 1 -- 172.21.15.17:0/95410622 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930011830 con 0x7fd9401051e0 2023-07-20T02:39:15.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd9367fc700 1 -- 172.21.15.17:0/95410622 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd930011e50 con 0x7fd9401051e0 2023-07-20T02:39:15.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.886+0000 7fd9367fc700 1 -- 172.21.15.17:0/95410622 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93000f2e0 con 0x7fd9401051e0 2023-07-20T02:39:15.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.887+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94019bf10 con 0x7fd9401051e0 2023-07-20T02:39:15.893 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.887+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9401a37b0 con 0x7fd9401051e0 2023-07-20T02:39:15.894 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.887+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94004e470 con 0x7fd9401051e0 2023-07-20T02:39:15.894 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.889+0000 7fd9367fc700 1 -- 172.21.15.17:0/95410622 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd930010380 con 0x7fd9401051e0 2023-07-20T02:39:15.894 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.889+0000 7fd9367fc700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd92c0a11d0 0x7fd92c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:15.895 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.889+0000 7fd945503700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd92c0a11d0 0x7fd92c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:15.895 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.889+0000 7fd9367fc700 1 -- 172.21.15.17:0/95410622 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd9300cbf70 con 0x7fd9401051e0 2023-07-20T02:39:15.895 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.890+0000 7fd945503700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd92c0a11d0 0x7fd92c0a3690 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd93c005950 tx=0x7fd93c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:15.895 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:15.893+0000 7fd9367fc700 1 -- 172.21.15.17:0/95410622 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd930095b60 con 0x7fd9401051e0 2023-07-20T02:39:15.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:15 smithi017 ceph-mon[127345]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:16.090 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.089+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9401a3aa0 con 0x7fd9401051e0 2023-07-20T02:39:16.091 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.089+0000 7fd9367fc700 1 -- 172.21.15.17:0/95410622 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd930099180 con 0x7fd9401051e0 2023-07-20T02:39:16.091 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2023-07-20T02:39:16.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd92c0a11d0 msgr2=0x7fd92c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:16.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd92c0a11d0 0x7fd92c0a3690 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd93c005950 tx=0x7fd93c0058e0 comp rx=0 tx=0).stop 2023-07-20T02:39:16.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9401051e0 msgr2=0x7fd940196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:16.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9401051e0 0x7fd940196e90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd93000b530 tx=0x7fd93000b840 comp rx=0 tx=0).stop 2023-07-20T02:39:16.093 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 shutdown_connections 2023-07-20T02:39:16.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd92c0a11d0 0x7fd92c0a3690 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:16.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd940105ba0 0x7fd94019b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:16.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd940068730 0x7fd940196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:16.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 --2- 172.21.15.17:0/95410622 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd9401051e0 0x7fd940196e90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:16.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.091+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 >> 172.21.15.17:0/95410622 conn(0x7fd940076770 msgr2=0x7fd9400fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:16.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.092+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 shutdown_connections 2023-07-20T02:39:16.094 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:16.092+0000 7fd947767700 1 -- 172.21.15.17:0/95410622 wait complete. 2023-07-20T02:39:16.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:15 smithi151 ceph-mon[129163]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:16.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:15 smithi146 ceph-mon[128841]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:16.514 INFO:tasks.cephadm:Adding mds.a on smithi017 2023-07-20T02:39:16.515 INFO:tasks.cephadm:Adding mds.d on smithi017 2023-07-20T02:39:16.515 INFO:tasks.cephadm:Adding mds.g on smithi017 2023-07-20T02:39:16.515 INFO:tasks.cephadm:Adding mds.j on smithi017 2023-07-20T02:39:16.515 INFO:tasks.cephadm:Adding mds.b on smithi146 2023-07-20T02:39:16.516 INFO:tasks.cephadm:Adding mds.e on smithi146 2023-07-20T02:39:16.516 INFO:tasks.cephadm:Adding mds.h on smithi146 2023-07-20T02:39:16.516 INFO:tasks.cephadm:Adding mds.k on smithi146 2023-07-20T02:39:16.516 INFO:tasks.cephadm:Adding mds.c on smithi151 2023-07-20T02:39:16.516 INFO:tasks.cephadm:Adding mds.f on smithi151 2023-07-20T02:39:16.517 INFO:tasks.cephadm:Adding mds.i on smithi151 2023-07-20T02:39:16.517 INFO:tasks.cephadm:Adding mds.l on smithi151 2023-07-20T02:39:16.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c -- ceph orch apply mds all '12;smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l' 2023-07-20T02:39:17.087 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:39:17.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:16 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/95410622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:17.287 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:16 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/95410622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:17.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:16 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/95410622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:18.139 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:17 smithi151 ceph-mon[129163]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:18.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:17 smithi017 ceph-mon[127345]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:18.297 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 -- 172.21.15.151:0/3081309003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a840ff9a0 msgr2=0x7f9a840ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:18.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 --2- 172.21.15.151:0/3081309003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a840ff9a0 0x7f9a840ffe20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009990 tx=0x7f9a74009ca0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 -- 172.21.15.151:0/3081309003 shutdown_connections 2023-07-20T02:39:18.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 --2- 172.21.15.151:0/3081309003 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a84100360 0x7f9a84107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 --2- 172.21.15.151:0/3081309003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a84100dd0 0x7f9a841011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 --2- 172.21.15.151:0/3081309003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a840ff9a0 0x7f9a840ffe20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 -- 172.21.15.151:0/3081309003 >> 172.21.15.151:0/3081309003 conn(0x7f9a840fa450 msgr2=0x7f9a840fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:18.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 -- 172.21.15.151:0/3081309003 shutdown_connections 2023-07-20T02:39:18.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.295+0000 7f9a89384700 1 -- 172.21.15.151:0/3081309003 wait complete. 2023-07-20T02:39:18.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 Processor -- start 2023-07-20T02:39:18.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 -- start start 2023-07-20T02:39:18.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a840ff9a0 0x7f9a84195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:18.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a84100360 0x7f9a84195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a84100dd0 0x7f9a8419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8419cfa0 con 0x7f9a840ff9a0 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8419d0e0 con 0x7f9a84100360 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a89384700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8419d270 con 0x7f9a84100dd0 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a84100dd0 0x7f9a8419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a84100dd0 0x7f9a8419c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:41230/0 (socket says 172.21.15.151:41230) 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 -- 172.21.15.151:0/3040401119 learned_addr learned my addr 172.21.15.151:0/3040401119 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-07-20T02:39:18.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a82ffd700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a840ff9a0 0x7f9a84195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a827fc700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a84100360 0x7f9a84195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 -- 172.21.15.151:0/3040401119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a84100360 msgr2=0x7f9a84195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a84100360 0x7f9a84195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 -- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a840ff9a0 msgr2=0x7f9a84195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a840ff9a0 0x7f9a84195410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.296+0000 7f9a837fe700 1 -- 172.21.15.151:0/3040401119 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740095f0 con 0x7f9a84100dd0 2023-07-20T02:39:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.297+0000 7f9a837fe700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a84100dd0 0x7f9a8419c9d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00add0 tx=0x7f9a7c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:18.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.297+0000 7f9a6bfff700 1 -- 172.21.15.151:0/3040401119 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c00f070 con 0x7f9a84100dd0 2023-07-20T02:39:18.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.297+0000 7f9a6bfff700 1 -- 172.21.15.151:0/3040401119 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a7c007b30 con 0x7f9a84100dd0 2023-07-20T02:39:18.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.297+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8419d550 con 0x7f9a84100dd0 2023-07-20T02:39:18.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.297+0000 7f9a6bfff700 1 -- 172.21.15.151:0/3040401119 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c0053e0 con 0x7f9a84100dd0 2023-07-20T02:39:18.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.297+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8419da00 con 0x7f9a84100dd0 2023-07-20T02:39:18.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.298+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8410afd0 con 0x7f9a84100dd0 2023-07-20T02:39:18.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.299+0000 7f9a6bfff700 1 -- 172.21.15.151:0/3040401119 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9a7c00c940 con 0x7f9a84100dd0 2023-07-20T02:39:18.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.299+0000 7f9a6bfff700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9a6c0a1150 0x7f9a6c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:18.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.300+0000 7f9a6bfff700 1 -- 172.21.15.151:0/3040401119 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9a7c0cbba0 con 0x7f9a84100dd0 2023-07-20T02:39:18.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.300+0000 7f9a82ffd700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9a6c0a1150 0x7f9a6c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:18.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.300+0000 7f9a82ffd700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9a6c0a1150 0x7f9a6c0a3610 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a78006fd0 tx=0x7f9a78008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:18.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.304+0000 7f9a6bfff700 1 -- 172.21.15.151:0/3040401119 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9a7c095820 con 0x7f9a84100dd0 2023-07-20T02:39:18.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:17 smithi146 ceph-mon[128841]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:18.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.500+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a84060cb0 con 0x7f9a6c0a1150 2023-07-20T02:39:18.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.509+0000 7f9a6bfff700 1 -- 172.21.15.151:0/3040401119 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f9a84060cb0 con 0x7f9a6c0a1150 2023-07-20T02:39:18.511 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mds.all update... 2023-07-20T02:39:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9a6c0a1150 msgr2=0x7f9a6c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9a6c0a1150 0x7f9a6c0a3610 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a78006fd0 tx=0x7f9a78008040 comp rx=0 tx=0).stop 2023-07-20T02:39:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a84100dd0 msgr2=0x7f9a8419c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a84100dd0 0x7f9a8419c9d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00add0 tx=0x7f9a7c00c330 comp rx=0 tx=0).stop 2023-07-20T02:39:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 shutdown_connections 2023-07-20T02:39:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f9a6c0a1150 0x7f9a6c0a3610 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a84100dd0 0x7f9a8419c9d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9a840ff9a0 0x7f9a84195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 --2- 172.21.15.151:0/3040401119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a84100360 0x7f9a84195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 >> 172.21.15.151:0/3040401119 conn(0x7f9a840fa450 msgr2=0x7f9a840fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 shutdown_connections 2023-07-20T02:39:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-07-20T02:39:18.511+0000 7f9a89384700 1 -- 172.21.15.151:0/3040401119 wait complete. 2023-07-20T02:39:18.950 DEBUG:teuthology.orchestra.run.smithi017:mds.a> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.a.service 2023-07-20T02:39:18.953 DEBUG:teuthology.orchestra.run.smithi017:mds.d> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.d.service 2023-07-20T02:39:18.991 INFO:journalctl@ceph.mds.a.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:39:18.992 DEBUG:teuthology.orchestra.run.smithi017:mds.g> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.g.service 2023-07-20T02:39:19.022 INFO:journalctl@ceph.mds.d.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:39:19.023 DEBUG:teuthology.orchestra.run.smithi017:mds.j> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.j.service 2023-07-20T02:39:19.052 INFO:journalctl@ceph.mds.g.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:39:19.053 DEBUG:teuthology.orchestra.run.smithi146:mds.b> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.b.service 2023-07-20T02:39:19.056 DEBUG:teuthology.orchestra.run.smithi146:mds.e> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.e.service 2023-07-20T02:39:19.080 INFO:journalctl@ceph.mds.j.smithi017.stdout:-- Logs begin at Thu 2023-07-20 02:09:51 UTC. -- 2023-07-20T02:39:19.095 INFO:journalctl@ceph.mds.b.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:39:19.096 DEBUG:teuthology.orchestra.run.smithi146:mds.h> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.h.service 2023-07-20T02:39:19.126 INFO:journalctl@ceph.mds.e.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:39:19.126 DEBUG:teuthology.orchestra.run.smithi146:mds.k> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.k.service 2023-07-20T02:39:19.154 INFO:journalctl@ceph.mds.h.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:39:19.155 DEBUG:teuthology.orchestra.run.smithi151:mds.c> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.c.service 2023-07-20T02:39:19.157 DEBUG:teuthology.orchestra.run.smithi151:mds.f> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.f.service 2023-07-20T02:39:19.181 INFO:journalctl@ceph.mds.k.smithi146.stdout:-- Logs begin at Thu 2023-07-20 02:10:15 UTC. -- 2023-07-20T02:39:19.187 INFO:journalctl@ceph.mds.c.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:39:19.188 DEBUG:teuthology.orchestra.run.smithi151:mds.i> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.i.service 2023-07-20T02:39:19.215 INFO:journalctl@ceph.mds.f.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:39:19.216 DEBUG:teuthology.orchestra.run.smithi151:mds.l> sudo journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.l.service 2023-07-20T02:39:19.243 INFO:journalctl@ceph.mds.i.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:39:19.244 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-20T02:39:19.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:39:19.269 INFO:journalctl@ceph.mds.l.smithi151.stdout:-- Logs begin at Thu 2023-07-20 02:10:13 UTC. -- 2023-07-20T02:39:19.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2999925612 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb4106440 msgr2=0x7fcfb41088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2999925612 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb4106440 0x7fcfb41088b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00b0a0 tx=0x7fcfac00b3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2999925612 shutdown_connections 2023-07-20T02:39:19.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2999925612 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb4106440 0x7fcfb41088b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2999925612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb41011e0 0x7fcfb41035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2999925612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb4103b10 0x7fcfb4105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2999925612 >> 172.21.15.17:0/2999925612 conn(0x7fcfb40fab90 msgr2=0x7fcfb40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:19.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2999925612 shutdown_connections 2023-07-20T02:39:19.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2999925612 wait complete. 2023-07-20T02:39:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 Processor -- start 2023-07-20T02:39:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.386+0000 7fcfbb8e6700 1 -- start start 2023-07-20T02:39:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb41011e0 0x7fcfb4100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb4103b10 0x7fcfb40ff190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb4106440 0x7fcfb40ff6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40ffe70 con 0x7fcfb41011e0 2023-07-20T02:39:19.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40fffe0 con 0x7fcfb4106440 2023-07-20T02:39:19.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb41a0590 con 0x7fcfb4103b10 2023-07-20T02:39:19.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb41011e0 0x7fcfb4100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb41011e0 0x7fcfb4100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43278/0 (socket says 172.21.15.17:43278) 2023-07-20T02:39:19.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 -- 172.21.15.17:0/2898192980 learned_addr learned my addr 172.21.15.17:0/2898192980 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:19.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9e83700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb4106440 0x7fcfb40ff6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb8e81700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb4103b10 0x7fcfb40ff190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 -- 172.21.15.17:0/2898192980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb4103b10 msgr2=0x7fcfb40ff190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb4103b10 0x7fcfb40ff190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 -- 172.21.15.17:0/2898192980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb4106440 msgr2=0x7fcfb40ff6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb4106440 0x7fcfb40ff6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 -- 172.21.15.17:0/2898192980 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac009d00 con 0x7fcfb41011e0 2023-07-20T02:39:19.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfb9682700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb41011e0 0x7fcfb4100b50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000b530 tx=0x7fcfb000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:19.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfaa7fc700 1 -- 172.21.15.17:0/2898192980 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0011830 con 0x7fcfb41011e0 2023-07-20T02:39:19.395 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfaa7fc700 1 -- 172.21.15.17:0/2898192980 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfb0011e50 con 0x7fcfb41011e0 2023-07-20T02:39:19.395 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb41a0870 con 0x7fcfb41011e0 2023-07-20T02:39:19.395 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfaa7fc700 1 -- 172.21.15.17:0/2898192980 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb000f2e0 con 0x7fcfb41011e0 2023-07-20T02:39:19.395 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.387+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb41a0e50 con 0x7fcfb41011e0 2023-07-20T02:39:19.395 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.388+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb40fc710 con 0x7fcfb41011e0 2023-07-20T02:39:19.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.389+0000 7fcfaa7fc700 1 -- 172.21.15.17:0/2898192980 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcfb00105a0 con 0x7fcfb41011e0 2023-07-20T02:39:19.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.390+0000 7fcfaa7fc700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa00a1220 0x7fcfa00a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.390+0000 7fcfb8e81700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa00a1220 0x7fcfa00a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.390+0000 7fcfaa7fc700 1 -- 172.21.15.17:0/2898192980 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcfb00cc2f0 con 0x7fcfb41011e0 2023-07-20T02:39:19.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.391+0000 7fcfb8e81700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa00a1220 0x7fcfa00a36e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4009c60 tx=0x7fcfa4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:19.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.393+0000 7fcfaa7fc700 1 -- 172.21.15.17:0/2898192980 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcfb0095ee0 con 0x7fcfb41011e0 2023-07-20T02:39:19.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.598+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfb402c970 con 0x7fcfb41011e0 2023-07-20T02:39:19.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.599+0000 7fcfaa7fc700 1 -- 172.21.15.17:0/2898192980 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18421 (secure 0 0 0) 0x7fcfb0099500 con 0x7fcfb41011e0 2023-07-20T02:39:19.601 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:19.601 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":79,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","created":"2023-07-20T02:32:50.175394+0000","modified":"2023-07-20T02:39:10.303454+0000","last_up_change":"2023-07-20T02:39:09.254013+0000","last_in_change":"2023-07-20T02:38:51.253472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:35:55.360132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb89ee2e-e4ff-4c83-9626-69cbfb526dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6803","nonce":795351069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6805","nonce":795351069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6809","nonce":795351069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6807","nonce":795351069}]},"public_addr":"172.21.15.17:6803/795351069","cluster_addr":"172.21.15.17:6805/795351069","heartbeat_back_addr":"172.21.15.17:6809/795351069","heartbeat_front_addr":"172.21.15.17:6807/795351069","state":["exists","up"]},{"osd":1,"uuid":"2d422389-0925-410c-bd28-50ad06d68cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6801","nonce":3123864696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6803","nonce":3123864696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6807","nonce":3123864696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6805","nonce":3123864696}]},"public_addr":"172.21.15.146:6801/3123864696","cluster_addr":"172.21.15.146:6803/3123864696","heartbeat_back_addr":"172.21.15.146:6807/3123864696","heartbeat_front_addr":"172.21.15.146:6805/3123864696","state":["exists","up"]},{"osd":2,"uuid":"a08d37e7-a3cd-4694-9441-b505bd167c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6801","nonce":1299864358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6803","nonce":1299864358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6807","nonce":1299864358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6805","nonce":1299864358}]},"public_addr":"172.21.15.151:6801/1299864358","cluster_addr":"172.21.15.151:6803/1299864358","heartbeat_back_addr":"172.21.15.151:6807/1299864358","heartbeat_front_addr":"172.21.15.151:6805/1299864358","state":["exists","up"]},{"osd":3,"uuid":"62ad61a1-05b0-4e38-a2eb-530485d806b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6811","nonce":604342850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6813","nonce":604342850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6817","nonce":604342850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6815","nonce":604342850}]},"public_addr":"172.21.15.17:6811/604342850","cluster_addr":"172.21.15.17:6813/604342850","heartbeat_back_addr":"172.21.15.17:6817/604342850","heartbeat_front_addr":"172.21.15.17:6815/604342850","state":["exists","up"]},{"osd":4,"uuid":"38ed288f-9c0d-415c-b52f-075d3b7599db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6809","nonce":4024595090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6811","nonce":4024595090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6815","nonce":4024595090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6813","nonce":4024595090}]},"public_addr":"172.21.15.146:6809/4024595090","cluster_addr":"172.21.15.146:6811/4024595090","heartbeat_back_addr":"172.21.15.146:6815/4024595090","heartbeat_front_addr":"172.21.15.146:6813/4024595090","state":["exists","up"]},{"osd":5,"uuid":"7da194cb-67fb-4c73-b84c-fa899c3635b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6809","nonce":1633219838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6811","nonce":1633219838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6815","nonce":1633219838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6813","nonce":1633219838}]},"public_addr":"172.21.15.151:6809/1633219838","cluster_addr":"172.21.15.151:6811/1633219838","heartbeat_back_addr":"172.21.15.151:6815/1633219838","heartbeat_front_addr":"172.21.15.151:6813/1633219838","state":["exists","up"]},{"osd":6,"uuid":"67ced483-d37a-46c1-97ae-288f28b091f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6819","nonce":3807389822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6821","nonce":3807389822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6825","nonce":3807389822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6823","nonce":3807389822}]},"public_addr":"172.21.15.17:6819/3807389822","cluster_addr":"172.21.15.17:6821/3807389822","heartbeat_back_addr":"172.21.15.17:6825/3807389822","heartbeat_front_addr":"172.21.15.17:6823/3807389822","state":["exists","up"]},{"osd":7,"uuid":"26199bbc-d82f-4e97-9218-0058426ef412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6817","nonce":708669673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6819","nonce":708669673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6823","nonce":708669673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6821","nonce":708669673}]},"public_addr":"172.21.15.146:6817/708669673","cluster_addr":"172.21.15.146:6819/708669673","heartbeat_back_addr":"172.21.15.146:6823/708669673","heartbeat_front_addr":"172.21.15.146:6821/708669673","state":["exists","up"]},{"osd":8,"uuid":"b1a33778-9ab5-45db-90ff-644794ee1cd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6817","nonce":2042667088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6819","nonce":2042667088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6823","nonce":2042667088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6821","nonce":2042667088}]},"public_addr":"172.21.15.151:6817/2042667088","cluster_addr":"172.21.15.151:6819/2042667088","heartbeat_back_addr":"172.21.15.151:6823/2042667088","heartbeat_front_addr":"172.21.15.151:6821/2042667088","state":["exists","up"]},{"osd":9,"uuid":"496d04b3-61a4-4405-aa52-5590aacf7129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6827","nonce":1403136476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6829","nonce":1403136476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6833","nonce":1403136476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6831","nonce":1403136476}]},"public_addr":"172.21.15.17:6827/1403136476","cluster_addr":"172.21.15.17:6829/1403136476","heartbeat_back_addr":"172.21.15.17:6833/1403136476","heartbeat_front_addr":"172.21.15.17:6831/1403136476","state":["exists","up"]},{"osd":10,"uuid":"6ee41a91-3ac3-4262-bff2-7531048859c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6825","nonce":2995236638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6827","nonce":2995236638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6831","nonce":2995236638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6829","nonce":2995236638}]},"public_addr":"172.21.15.146:6825/2995236638","cluster_addr":"172.21.15.146:6827/2995236638","heartbeat_back_addr":"172.21.15.146:6831/2995236638","heartbeat_front_addr":"172.21.15.146:6829/2995236638","state":["exists","up"]},{"osd":11,"uuid":"282d07a4-1bb0-4239-9c1e-adeebe9d7432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6825","nonce":1663985079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6827","nonce":1663985079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6831","nonce":1663985079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6829","nonce":1663985079}]},"public_addr":"172.21.15.151:6825/1663985079","cluster_addr":"172.21.15.151:6827/1663985079","heartbeat_back_addr":"172.21.15.151:6831/1663985079","heartbeat_front_addr":"172.21.15.151:6829/1663985079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:07.560935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:29.818964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:51.802583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:12.985262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:34.792736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:56.378329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:17.498840+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:40.421290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:02.192805+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:23.342606+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:45.695709+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:39:07.701215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2979320785":"2023-07-21T02:33:44.823257+0000","172.21.15.17:0/979357839":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6800/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/714275157":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1888486288":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/910909892":"2023-07-21T02:33:19.367142+0000","172.21.15.17:6800/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1536507180":"2023-07-21T02:33:44.823257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:39:19.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa00a1220 msgr2=0x7fcfa00a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa00a1220 0x7fcfa00a36e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4009c60 tx=0x7fcfa4009260 comp rx=0 tx=0).stop 2023-07-20T02:39:19.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb41011e0 msgr2=0x7fcfb4100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb41011e0 0x7fcfb4100b50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000b530 tx=0x7fcfb000b840 comp rx=0 tx=0).stop 2023-07-20T02:39:19.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 shutdown_connections 2023-07-20T02:39:19.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa00a1220 0x7fcfa00a36e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.604 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb4106440 0x7fcfb40ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.604 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb41011e0 0x7fcfb4100b50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.604 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 --2- 172.21.15.17:0/2898192980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb4103b10 0x7fcfb40ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.604 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 >> 172.21.15.17:0/2898192980 conn(0x7fcfb40fab90 msgr2=0x7fcfb4101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:19.604 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 shutdown_connections 2023-07-20T02:39:19.604 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.601+0000 7fcfbb8e6700 1 -- 172.21.15.17:0/2898192980 wait complete. 2023-07-20T02:39:19.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:19.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: Saving service mds.all spec with placement smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l;count:12 2023-07-20T02:39:19.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:19.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:19.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:19.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:19.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:19.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:19.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:19.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:19.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:19 smithi017 ceph-mon[127345]: Deploying daemon mds.l on smithi151 2023-07-20T02:39:19.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:35:55.360132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T02:39:19.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T02:39:19.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 -- 172.21.15.17:0/379500589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd0100850 msgr2=0x7fbbd0100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/379500589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd0100850 0x7fbbd0100cb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0099c0 tx=0x7fbbbc009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 -- 172.21.15.17:0/379500589 shutdown_connections 2023-07-20T02:39:19.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/379500589 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbbd01011f0 0x7fbbd0107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.751 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/379500589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbd00ff660 0x7fbbd00ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/379500589 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd0100850 0x7fbbd0100cb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 -- 172.21.15.17:0/379500589 >> 172.21.15.17:0/379500589 conn(0x7fbbd00fabd0 msgr2=0x7fbbd00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:19.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 -- 172.21.15.17:0/379500589 shutdown_connections 2023-07-20T02:39:19.752 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.749+0000 7fbbd6afa700 1 -- 172.21.15.17:0/379500589 wait complete. 2023-07-20T02:39:19.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 Processor -- start 2023-07-20T02:39:19.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 -- start start 2023-07-20T02:39:19.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbbd00ff660 0x7fbbd0068460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbd0100850 0x7fbbd019b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.754 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd01011f0 0x7fbbd019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd019dfe0 con 0x7fbbd01011f0 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd019e150 con 0x7fbbd0100850 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd6afa700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd019e310 con 0x7fbbd00ff660 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd01011f0 0x7fbbd019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd01011f0 0x7fbbd019d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43296/0 (socket says 172.21.15.17:43296) 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 -- 172.21.15.17:0/2247391615 learned_addr learned my addr 172.21.15.17:0/2247391615 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbcffff700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbd0100850 0x7fbbd019b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd4896700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbbd00ff660 0x7fbbd0068460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 -- 172.21.15.17:0/2247391615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbbd00ff660 msgr2=0x7fbbd0068460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbbd00ff660 0x7fbbd0068460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 -- 172.21.15.17:0/2247391615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbd0100850 msgr2=0x7fbbd019b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbd0100850 0x7fbbd019b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 -- 172.21.15.17:0/2247391615 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc0095f0 con 0x7fbbd01011f0 2023-07-20T02:39:19.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.750+0000 7fbbd5097700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd01011f0 0x7fbbd019d810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8007ee0 tx=0x7fbbc800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:19.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.751+0000 7fbbcdffb700 1 -- 172.21.15.17:0/2247391615 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc800ddd0 con 0x7fbbd01011f0 2023-07-20T02:39:19.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.751+0000 7fbbcdffb700 1 -- 172.21.15.17:0/2247391615 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbc800f040 con 0x7fbbd01011f0 2023-07-20T02:39:19.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.751+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd019e5f0 con 0x7fbbd01011f0 2023-07-20T02:39:19.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.751+0000 7fbbcdffb700 1 -- 172.21.15.17:0/2247391615 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc80048e0 con 0x7fbbd01011f0 2023-07-20T02:39:19.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.751+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd019ea80 con 0x7fbbd01011f0 2023-07-20T02:39:19.757 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.752+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd010ae70 con 0x7fbbd01011f0 2023-07-20T02:39:19.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.753+0000 7fbbcdffb700 1 -- 172.21.15.17:0/2247391615 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbbc8003a00 con 0x7fbbd01011f0 2023-07-20T02:39:19.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.753+0000 7fbbcdffb700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbbc00a1150 0x7fbbc00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:19.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.754+0000 7fbbd4896700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbbc00a1150 0x7fbbc00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:19.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.754+0000 7fbbcdffb700 1 -- 172.21.15.17:0/2247391615 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbbc80cbe60 con 0x7fbbd01011f0 2023-07-20T02:39:19.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.754+0000 7fbbd4896700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbbc00a1150 0x7fbbc00a3610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4005950 tx=0x7fbbc40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:19.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.758+0000 7fbbcdffb700 1 -- 172.21.15.17:0/2247391615 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbbc804e780 con 0x7fbbd01011f0 2023-07-20T02:39:19.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:19.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: Saving service mds.all spec with placement smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l;count:12 2023-07-20T02:39:19.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:19.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:19.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:19.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:19.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:19.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:19.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:19.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:19.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:19 smithi146 ceph-mon[128841]: Deploying daemon mds.l on smithi151 2023-07-20T02:39:19.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:19.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: Saving service mds.all spec with placement smithi017=a;smithi017=d;smithi017=g;smithi017=j;smithi146=b;smithi146=e;smithi146=h;smithi146=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l;count:12 2023-07-20T02:39:19.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:19.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:19.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:19.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:19.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:19.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:19.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:19.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:19.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:19 smithi151 ceph-mon[129163]: Deploying daemon mds.l on smithi151 2023-07-20T02:39:19.959 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.957+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbbd0106610 con 0x7fbbd01011f0 2023-07-20T02:39:19.959 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.958+0000 7fbbcdffb700 1 -- 172.21.15.17:0/2247391615 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbbc80959b0 con 0x7fbbd01011f0 2023-07-20T02:39:19.960 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2023-07-20T02:39:19.961 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbbc00a1150 msgr2=0x7fbbc00a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbbc00a1150 0x7fbbc00a3610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4005950 tx=0x7fbbc40058e0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd01011f0 msgr2=0x7fbbd019d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:19.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd01011f0 0x7fbbd019d810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8007ee0 tx=0x7fbbc800d3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 shutdown_connections 2023-07-20T02:39:19.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbbc00a1150 0x7fbbc00a3610 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbbd01011f0 0x7fbbd019d810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.962 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbbd00ff660 0x7fbbd0068460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 --2- 172.21.15.17:0/2247391615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbd0100850 0x7fbbd019b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:19.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 >> 172.21.15.17:0/2247391615 conn(0x7fbbd00fabd0 msgr2=0x7fbbd00fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:19.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 shutdown_connections 2023-07-20T02:39:19.963 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:19.960+0000 7fbbd6afa700 1 -- 172.21.15.17:0/2247391615 wait complete. 2023-07-20T02:39:19.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:20.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 -- 172.21.15.17:0/1756953391 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec0ff6a0 msgr2=0x7f6dec0ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:20.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 --2- 172.21.15.17:0/1756953391 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec0ff6a0 0x7f6dec0ffac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0099c0 tx=0x7f6ddc009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 -- 172.21.15.17:0/1756953391 shutdown_connections 2023-07-20T02:39:20.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 --2- 172.21.15.17:0/1756953391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dec101230 0x7f6dec107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.100 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 --2- 172.21.15.17:0/1756953391 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec0ff6a0 0x7f6dec0ffac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 --2- 172.21.15.17:0/1756953391 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dec100890 0x7f6dec100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 -- 172.21.15.17:0/1756953391 >> 172.21.15.17:0/1756953391 conn(0x7f6dec0fabf0 msgr2=0x7f6dec0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:20.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 -- 172.21.15.17:0/1756953391 shutdown_connections 2023-07-20T02:39:20.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.098+0000 7f6df1890700 1 -- 172.21.15.17:0/1756953391 wait complete. 2023-07-20T02:39:20.101 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 Processor -- start 2023-07-20T02:39:20.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 -- start start 2023-07-20T02:39:20.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dec0ff6a0 0x7f6dec196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:20.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dec100890 0x7f6dec1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:20.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec101230 0x7f6dec19da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:20.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec19e000 con 0x7f6dec101230 2023-07-20T02:39:20.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec19e140 con 0x7f6dec100890 2023-07-20T02:39:20.103 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6df1890700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec19e2d0 con 0x7f6dec0ff6a0 2023-07-20T02:39:20.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec101230 0x7f6dec19da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:20.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec101230 0x7f6dec19da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43308/0 (socket says 172.21.15.17:43308) 2023-07-20T02:39:20.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deaffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dec0ff6a0 0x7f6dec196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:20.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 -- 172.21.15.17:0/3815117765 learned_addr learned my addr 172.21.15.17:0/3815117765 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:20.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6dea7fc700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dec100890 0x7f6dec1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:20.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 -- 172.21.15.17:0/3815117765 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dec0ff6a0 msgr2=0x7f6dec196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:20.104 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dec0ff6a0 0x7f6dec196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 -- 172.21.15.17:0/3815117765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dec100890 msgr2=0x7f6dec1969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dec100890 0x7f6dec1969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.099+0000 7f6deb7fe700 1 -- 172.21.15.17:0/3815117765 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc0095f0 con 0x7f6dec101230 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.100+0000 7f6deb7fe700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec101230 0x7f6dec19da30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6de400eb80 tx=0x7f6de400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.100+0000 7f6df088e700 1 -- 172.21.15.17:0/3815117765 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4018070 con 0x7f6dec101230 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.100+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dec19e550 con 0x7f6dec101230 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.100+0000 7f6df088e700 1 -- 172.21.15.17:0/3815117765 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6de400f040 con 0x7f6dec101230 2023-07-20T02:39:20.105 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.100+0000 7f6df088e700 1 -- 172.21.15.17:0/3815117765 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de400cde0 con 0x7f6dec101230 2023-07-20T02:39:20.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.100+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dec19ea50 con 0x7f6dec101230 2023-07-20T02:39:20.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.101+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dec10aeb0 con 0x7f6dec101230 2023-07-20T02:39:20.106 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.102+0000 7f6df088e700 1 -- 172.21.15.17:0/3815117765 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6de4003a50 con 0x7f6dec101230 2023-07-20T02:39:20.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.103+0000 7f6df088e700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6dd40a11a0 0x7f6dd40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:20.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.103+0000 7f6deaffd700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6dd40a11a0 0x7f6dd40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:20.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.103+0000 7f6df088e700 1 -- 172.21.15.17:0/3815117765 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6de40cbfe0 con 0x7f6dec101230 2023-07-20T02:39:20.107 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.103+0000 7f6deaffd700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6dd40a11a0 0x7f6dd40a3660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009990 tx=0x7f6ddc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:20.108 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.107+0000 7f6df088e700 1 -- 172.21.15.17:0/3815117765 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6de4095c60 con 0x7f6dec101230 2023-07-20T02:39:20.112 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:39:20 smithi151 systemd[1]: Starting Ceph mds.l for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:20.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.350+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dec19f090 con 0x7f6dec101230 2023-07-20T02:39:20.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.351+0000 7f6df088e700 1 -- 172.21.15.17:0/3815117765 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f6de4084090 con 0x7f6dec101230 2023-07-20T02:39:20.353 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:20.353 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-20T02:39:20.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6dd40a11a0 msgr2=0x7f6dd40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:20.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6dd40a11a0 0x7f6dd40a3660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009990 tx=0x7f6ddc005fb0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec101230 msgr2=0x7f6dec19da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:20.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec101230 0x7f6dec19da30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6de400eb80 tx=0x7f6de400ee90 comp rx=0 tx=0).stop 2023-07-20T02:39:20.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 shutdown_connections 2023-07-20T02:39:20.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f6dd40a11a0 0x7f6dd40a3660 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f6dec101230 0x7f6dec19da30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dec0ff6a0 0x7f6dec196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 --2- 172.21.15.17:0/3815117765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dec100890 0x7f6dec1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:20.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 >> 172.21.15.17:0/3815117765 conn(0x7f6dec0fabf0 msgr2=0x7f6dec068b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:20.356 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 shutdown_connections 2023-07-20T02:39:20.356 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:20.353+0000 7f6df1890700 1 -- 172.21.15.17:0/3815117765 wait complete. 2023-07-20T02:39:20.356 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 1 2023-07-20T02:39:20.620 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:39:20 smithi151 podman[154401]: 2023-07-20T02:39:20.621 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:20 smithi151 ceph-mon[129163]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:20.621 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:20 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2898192980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:20.621 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:20 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2247391615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:20.621 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:20 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3815117765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:20.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:20 smithi017 ceph-mon[127345]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:20.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:20 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2898192980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:20.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:20 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2247391615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:20.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:20 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3815117765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:20.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:20 smithi146 ceph-mon[128841]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:20.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:20 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2898192980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:20.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:20 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2247391615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:20.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:20 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3815117765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:21.112 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:39:20 smithi151 ceph-mds[154441]: -- 172.21.15.151:0/1773511302 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557aaebd51e0 con 0x557aaebaac00 2023-07-20T02:39:21.112 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:39:20 smithi151 bash[154401]: 2a252a16ae3927af48303ff9c6bd09c9ef69211bc88a98cee58557f593d95ebe 2023-07-20T02:39:21.112 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:39:20 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-l[154417]: starting mds.l at 2023-07-20T02:39:21.113 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:39:20 smithi151 systemd[1]: Started Ceph mds.l for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:21.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.151:6832/3119401116,v1:172.21.15.151:6833/3119401116] up:boot 2023-07-20T02:39:21.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: fsmap 1 up:standby 2023-07-20T02:39:21.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T02:39:21.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:21.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:21.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:21.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:21.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:21.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:21.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:21 smithi146 ceph-mon[128841]: Deploying daemon mds.k on smithi146 2023-07-20T02:39:21.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.151:6832/3119401116,v1:172.21.15.151:6833/3119401116] up:boot 2023-07-20T02:39:21.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: fsmap 1 up:standby 2023-07-20T02:39:21.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T02:39:21.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:21.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:21.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:21.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:21.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:21.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:21.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:21 smithi017 ceph-mon[127345]: Deploying daemon mds.k on smithi146 2023-07-20T02:39:22.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.151:6832/3119401116,v1:172.21.15.151:6833/3119401116] up:boot 2023-07-20T02:39:22.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: fsmap 1 up:standby 2023-07-20T02:39:22.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T02:39:22.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:22.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:22.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:22.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:22.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:22.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:22.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:21 smithi151 ceph-mon[129163]: Deploying daemon mds.k on smithi146 2023-07-20T02:39:22.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:22.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 -- 172.21.15.17:0/1778410205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 msgr2=0x7f5184101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:22.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1778410205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184101de0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f51780099c0 tx=0x7f5178009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 -- 172.21.15.17:0/1778410205 shutdown_connections 2023-07-20T02:39:22.508 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1778410205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184102320 0x7f5184108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1778410205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5184100790 0x7f5184100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1778410205 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184101de0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 -- 172.21.15.17:0/1778410205 >> 172.21.15.17:0/1778410205 conn(0x7f51840fbd20 msgr2=0x7f51840fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:22.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 -- 172.21.15.17:0/1778410205 shutdown_connections 2023-07-20T02:39:22.509 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.506+0000 7f5188ecd700 1 -- 172.21.15.17:0/1778410205 wait complete. 2023-07-20T02:39:22.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 Processor -- start 2023-07-20T02:39:22.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 -- start start 2023-07-20T02:39:22.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184100790 0x7f5184198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:22.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:22.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5184102320 0x7f518419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:22.511 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51841a0100 con 0x7f5184101980 2023-07-20T02:39:22.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51841a0240 con 0x7f5184100790 2023-07-20T02:39:22.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5188ecd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51841a03d0 con 0x7f5184102320 2023-07-20T02:39:22.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5181d9b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:22.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5181d9b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43320/0 (socket says 172.21.15.17:43320) 2023-07-20T02:39:22.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5181d9b700 1 -- 172.21.15.17:0/1958399801 learned_addr learned my addr 172.21.15.17:0/1958399801 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:22.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f518259c700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184100790 0x7f5184198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:22.512 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.507+0000 7f5182d9d700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5184102320 0x7f518419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5181d9b700 1 -- 172.21.15.17:0/1958399801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5184102320 msgr2=0x7f518419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5181d9b700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5184102320 0x7f518419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5181d9b700 1 -- 172.21.15.17:0/1958399801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184100790 msgr2=0x7f5184198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5181d9b700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184100790 0x7f5184198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5181d9b700 1 -- 172.21.15.17:0/1958399801 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780095f0 con 0x7f5184101980 2023-07-20T02:39:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5181d9b700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184198ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5178009990 tx=0x7f5178004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f51737fe700 1 -- 172.21.15.17:0/1958399801 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517801d070 con 0x7f5184101980 2023-07-20T02:39:22.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f51737fe700 1 -- 172.21.15.17:0/1958399801 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5178004010 con 0x7f5184101980 2023-07-20T02:39:22.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51841a0650 con 0x7f5184101980 2023-07-20T02:39:22.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f51737fe700 1 -- 172.21.15.17:0/1958399801 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178022d40 con 0x7f5184101980 2023-07-20T02:39:22.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.508+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51841a0b00 con 0x7f5184101980 2023-07-20T02:39:22.514 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.509+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518410bfb0 con 0x7f5184101980 2023-07-20T02:39:22.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.510+0000 7f51737fe700 1 -- 172.21.15.17:0/1958399801 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5178022470 con 0x7f5184101980 2023-07-20T02:39:22.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.511+0000 7f51737fe700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f516c0a11a0 0x7f516c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:22.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.511+0000 7f518259c700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f516c0a11a0 0x7f516c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:22.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.511+0000 7f51737fe700 1 -- 172.21.15.17:0/1958399801 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f51780cdb60 con 0x7f5184101980 2023-07-20T02:39:22.515 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.511+0000 7f518259c700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f516c0a11a0 0x7f516c0a3660 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f51841017e0 tx=0x7f517400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:22.516 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.514+0000 7f51737fe700 1 -- 172.21.15.17:0/1958399801 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f51780977e0 con 0x7f5184101980 2023-07-20T02:39:22.728 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:39:22 smithi146 systemd[1]: Starting Ceph mds.k for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:22.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.758+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f518410c1d0 con 0x7f5184101980 2023-07-20T02:39:22.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.758+0000 7f51737fe700 1 -- 172.21.15.17:0/1958399801 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f517809ae00 con 0x7f5184101980 2023-07-20T02:39:22.760 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:22.761 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T02:39:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f516c0a11a0 msgr2=0x7f516c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f516c0a11a0 0x7f516c0a3660 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f51841017e0 tx=0x7f517400b410 comp rx=0 tx=0).stop 2023-07-20T02:39:22.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 msgr2=0x7f5184198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184198ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5178009990 tx=0x7f5178004730 comp rx=0 tx=0).stop 2023-07-20T02:39:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 shutdown_connections 2023-07-20T02:39:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f516c0a11a0 0x7f516c0a3660 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5184102320 0x7f518419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184100790 0x7f5184198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 --2- 172.21.15.17:0/1958399801 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5184101980 0x7f5184198ac0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:22.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 >> 172.21.15.17:0/1958399801 conn(0x7f51840fbd20 msgr2=0x7f51841071a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 shutdown_connections 2023-07-20T02:39:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:22.760+0000 7f5188ecd700 1 -- 172.21.15.17:0/1958399801 wait complete. 2023-07-20T02:39:22.764 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 2 2023-07-20T02:39:23.034 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:22 smithi146 ceph-mon[128841]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:23.035 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:39:22 smithi146 podman[153113]: 2023-07-20T02:39:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:22 smithi151 ceph-mon[129163]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:23.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:22 smithi017 ceph-mon[127345]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:23.318 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mds[153153]: -- 172.21.15.146:0/3823302800 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564250b351e0 con 0x564250b0ac00 2023-07-20T02:39:23.319 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:39:23 smithi146 bash[153113]: 287fc5112196394bd690d06ce09672abbf838678c81cacd4e5dedf3bf11cbfbe 2023-07-20T02:39:23.319 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-k[153129]: starting mds.k at 2023-07-20T02:39:23.319 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:39:23 smithi146 systemd[1]: Started Ceph mds.k for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:23.985 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:23 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1958399801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:23.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:23 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:23.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:23 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:23.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:23 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:23.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:23 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:23.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:23 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:23.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:23 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:24.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1958399801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:24.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:24.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:24.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:24.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:24.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:24.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:23 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:24.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:23 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1958399801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:24.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:23 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:24.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:23 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:24.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:23 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:24.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:23 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:24.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:23 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:24.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:23 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:24.775 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:24.930 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2340417956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae81011e0 msgr2=0x7f1ae8107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:24.930 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2340417956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae81011e0 0x7f1ae8107a90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000b0d0 tx=0x7f1ae000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:24.930 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2340417956 shutdown_connections 2023-07-20T02:39:24.930 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2340417956 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae81011e0 0x7f1ae8107a90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:24.931 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2340417956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ae80ff650 0x7f1ae80ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:24.931 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2340417956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae8100840 0x7f1ae8100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:24.931 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2340417956 >> 172.21.15.17:0/2340417956 conn(0x7f1ae80faba0 msgr2=0x7f1ae80fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:24.931 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2340417956 shutdown_connections 2023-07-20T02:39:24.931 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.928+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2340417956 wait complete. 2023-07-20T02:39:24.933 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 Processor -- start 2023-07-20T02:39:24.933 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 -- start start 2023-07-20T02:39:24.933 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ae80ff650 0x7f1ae8103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:24.933 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae8100840 0x7f1ae8101e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:24.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae81011e0 0x7f1ae81023d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:24.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8102b70 con 0x7f1ae8100840 2023-07-20T02:39:24.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8102ce0 con 0x7f1ae81011e0 2023-07-20T02:39:24.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aeeb50700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae81a0550 con 0x7f1ae80ff650 2023-07-20T02:39:24.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae8100840 0x7f1ae8101e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:24.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae8100840 0x7f1ae8101e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43332/0 (socket says 172.21.15.17:43332) 2023-07-20T02:39:24.934 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1ae7fff700 1 -- 172.21.15.17:0/2583874789 learned_addr learned my addr 172.21.15.17:0/2583874789 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aec8ec700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ae80ff650 0x7f1ae8103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.929+0000 7f1aed0ed700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae81011e0 0x7f1ae81023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae7fff700 1 -- 172.21.15.17:0/2583874789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ae80ff650 msgr2=0x7f1ae8103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae7fff700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ae80ff650 0x7f1ae8103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae7fff700 1 -- 172.21.15.17:0/2583874789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae81011e0 msgr2=0x7f1ae81023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae7fff700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae81011e0 0x7f1ae81023d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae7fff700 1 -- 172.21.15.17:0/2583874789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae0009d00 con 0x7f1ae8100840 2023-07-20T02:39:24.935 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1aed0ed700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae81011e0 0x7f1ae81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:39:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae7fff700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae8100840 0x7f1ae8101e90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c920 tx=0x7f1ad400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae5ffb700 1 -- 172.21.15.17:0/2583874789 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4007b00 con 0x7f1ae8100840 2023-07-20T02:39:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae5ffb700 1 -- 172.21.15.17:0/2583874789 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ad400ce70 con 0x7f1ae8100840 2023-07-20T02:39:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae81a0830 con 0x7f1ae8100840 2023-07-20T02:39:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1ae5ffb700 1 -- 172.21.15.17:0/2583874789 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad40054e0 con 0x7f1ae8100840 2023-07-20T02:39:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.930+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae81a0d90 con 0x7f1ae8100840 2023-07-20T02:39:24.936 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.931+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae8107690 con 0x7f1ae8100840 2023-07-20T02:39:24.937 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.932+0000 7f1ae5ffb700 1 -- 172.21.15.17:0/2583874789 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1ad4004750 con 0x7f1ae8100840 2023-07-20T02:39:24.937 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.933+0000 7f1ae5ffb700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1ad80a1220 0x7f1ad80a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:24.937 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.933+0000 7f1aec8ec700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1ad80a1220 0x7f1ad80a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:24.938 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.933+0000 7f1ae5ffb700 1 -- 172.21.15.17:0/2583874789 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1ad40cbf50 con 0x7f1ae8100840 2023-07-20T02:39:24.938 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.933+0000 7f1aec8ec700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1ad80a1220 0x7f1ad80a36e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1ae81006a0 tx=0x7f1adc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:24.939 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:24.938+0000 7f1ae5ffb700 1 -- 172.21.15.17:0/2583874789 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1ad4095b40 con 0x7f1ae8100840 2023-07-20T02:39:24.954 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:39:24 smithi017 systemd[1]: Starting Ceph mds.j for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:24 smithi017 ceph-mon[127345]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:24 smithi017 ceph-mon[127345]: Deploying daemon mds.j on smithi017 2023-07-20T02:39:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:24 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.146:6832/1472683577,v1:172.21.15.146:6833/1472683577] up:boot 2023-07-20T02:39:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:24 smithi017 ceph-mon[127345]: fsmap 2 up:standby 2023-07-20T02:39:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:24 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T02:39:25.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:24 smithi146 ceph-mon[128841]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:25.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:24 smithi146 ceph-mon[128841]: Deploying daemon mds.j on smithi017 2023-07-20T02:39:25.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:24 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.146:6832/1472683577,v1:172.21.15.146:6833/1472683577] up:boot 2023-07-20T02:39:25.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:24 smithi146 ceph-mon[128841]: fsmap 2 up:standby 2023-07-20T02:39:25.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:24 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T02:39:25.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:24 smithi151 ceph-mon[129163]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:25.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:24 smithi151 ceph-mon[129163]: Deploying daemon mds.j on smithi017 2023-07-20T02:39:25.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:24 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.146:6832/1472683577,v1:172.21.15.146:6833/1472683577] up:boot 2023-07-20T02:39:25.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:24 smithi151 ceph-mon[129163]: fsmap 2 up:standby 2023-07-20T02:39:25.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:24 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T02:39:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.193+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ae802c970 con 0x7f1ae8100840 2023-07-20T02:39:25.195 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.193+0000 7f1ae5ffb700 1 -- 172.21.15.17:0/2583874789 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f1ad401d800 con 0x7f1ae8100840 2023-07-20T02:39:25.195 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:25.195 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T02:39:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1ad80a1220 msgr2=0x7f1ad80a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1ad80a1220 0x7f1ad80a36e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1ae81006a0 tx=0x7f1adc00b410 comp rx=0 tx=0).stop 2023-07-20T02:39:25.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae8100840 msgr2=0x7f1ae8101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae8100840 0x7f1ae8101e90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c920 tx=0x7f1ad400cc30 comp rx=0 tx=0).stop 2023-07-20T02:39:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 shutdown_connections 2023-07-20T02:39:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1ad80a1220 0x7f1ad80a36e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae81011e0 0x7f1ae81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1ae80ff650 0x7f1ae8103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 --2- 172.21.15.17:0/2583874789 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1ae8100840 0x7f1ae8101e90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:25.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.195+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 >> 172.21.15.17:0/2583874789 conn(0x7f1ae80faba0 msgr2=0x7f1ae80fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:25.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.196+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 shutdown_connections 2023-07-20T02:39:25.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:25.196+0000 7f1aeeb50700 1 -- 172.21.15.17:0/2583874789 wait complete. 2023-07-20T02:39:25.199 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 3 2023-07-20T02:39:25.302 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:39:25 smithi017 podman[160641]: 2023-07-20T02:39:25.303 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:39:25 smithi017 bash[160641]: 6ee122a1909f372644279c6eedca81e7f76a440d1cc48800f141a251e0952ac7 2023-07-20T02:39:25.303 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mds[160703]: -- 172.21.15.17:0/1811769211 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8672951e0 con 0x55a86726a800 2023-07-20T02:39:25.303 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-j[160677]: starting mds.j at 2023-07-20T02:39:25.705 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:39:25 smithi017 systemd[1]: Started Ceph mds.j for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:26.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:26.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2583874789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:26.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:26.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:26.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:26.196 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:25 smithi017 ceph-mon[127345]: Deploying daemon mds.d on smithi017 2023-07-20T02:39:26.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:26.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2583874789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:26.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:26.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:26.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:26.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:25 smithi146 ceph-mon[128841]: Deploying daemon mds.d on smithi017 2023-07-20T02:39:26.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:26.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2583874789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:26.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:26.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:26.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:26.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:26.364 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:25 smithi151 ceph-mon[129163]: Deploying daemon mds.d on smithi017 2023-07-20T02:39:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:26 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.17:6834/2807536449,v1:172.21.15.17:6835/2807536449] up:boot 2023-07-20T02:39:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:26 smithi017 ceph-mon[127345]: fsmap 3 up:standby 2023-07-20T02:39:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:26 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T02:39:27.206 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:39:26 smithi017 systemd[1]: Starting Ceph mds.d for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:27.210 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:27.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:26 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.17:6834/2807536449,v1:172.21.15.17:6835/2807536449] up:boot 2023-07-20T02:39:27.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:26 smithi146 ceph-mon[128841]: fsmap 3 up:standby 2023-07-20T02:39:27.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:26 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T02:39:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 -- 172.21.15.17:0/4229344747 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 msgr2=0x7f32c4107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:27.347 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/4229344747 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c4107a90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00b0d0 tx=0x7f32bc00b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 -- 172.21.15.17:0/4229344747 shutdown_connections 2023-07-20T02:39:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/4229344747 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c4107a90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/4229344747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32c40ff650 0x7f32c40ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/4229344747 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c4100840 0x7f32c4100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 -- 172.21.15.17:0/4229344747 >> 172.21.15.17:0/4229344747 conn(0x7f32c40faba0 msgr2=0x7f32c40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:27.348 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 -- 172.21.15.17:0/4229344747 shutdown_connections 2023-07-20T02:39:27.349 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.345+0000 7f32c9fb1700 1 -- 172.21.15.17:0/4229344747 wait complete. 2023-07-20T02:39:27.350 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.346+0000 7f32c9fb1700 1 Processor -- start 2023-07-20T02:39:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.346+0000 7f32c9fb1700 1 -- start start 2023-07-20T02:39:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c9fb1700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32c40ff650 0x7f32c41953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c9fb1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c4100840 0x7f32c4195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c9fb1700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c9fb1700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c419cf50 con 0x7f32c41011e0 2023-07-20T02:39:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c9fb1700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c419d090 con 0x7f32c40ff650 2023-07-20T02:39:27.351 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c9fb1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c419d220 con 0x7f32c4100840 2023-07-20T02:39:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43358/0 (socket says 172.21.15.17:43358) 2023-07-20T02:39:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 -- 172.21.15.17:0/3003436208 learned_addr learned my addr 172.21.15.17:0/3003436208 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c37fe700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32c40ff650 0x7f32c41953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c2ffd700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c4100840 0x7f32c4195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 -- 172.21.15.17:0/3003436208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c4100840 msgr2=0x7f32c4195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c4100840 0x7f32c4195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 -- 172.21.15.17:0/3003436208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32c40ff650 msgr2=0x7f32c41953d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32c40ff650 0x7f32c41953d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 -- 172.21.15.17:0/3003436208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc009d00 con 0x7f32c41011e0 2023-07-20T02:39:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c3fff700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c419c980 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f32bc007a00 tx=0x7f32bc0094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.347+0000 7f32c0ff9700 1 -- 172.21.15.17:0/3003436208 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc00e040 con 0x7f32c41011e0 2023-07-20T02:39:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.348+0000 7f32c0ff9700 1 -- 172.21.15.17:0/3003436208 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32bc003e10 con 0x7f32c41011e0 2023-07-20T02:39:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.348+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c419d4a0 con 0x7f32c41011e0 2023-07-20T02:39:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.348+0000 7f32c0ff9700 1 -- 172.21.15.17:0/3003436208 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc012a90 con 0x7f32c41011e0 2023-07-20T02:39:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.348+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c419d870 con 0x7f32c41011e0 2023-07-20T02:39:27.354 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.349+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c41064e0 con 0x7f32c41011e0 2023-07-20T02:39:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.350+0000 7f32c0ff9700 1 -- 172.21.15.17:0/3003436208 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f32bc0056c0 con 0x7f32c41011e0 2023-07-20T02:39:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.350+0000 7f32c0ff9700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f32b00a11a0 0x7f32b00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.350+0000 7f32c0ff9700 1 -- 172.21.15.17:0/3003436208 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f32bc0cdc50 con 0x7f32c41011e0 2023-07-20T02:39:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.350+0000 7f32c37fe700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f32b00a11a0 0x7f32b00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.351+0000 7f32c37fe700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f32b00a11a0 0x7f32b00a3660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f32c41006a0 tx=0x7f32ac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:27.356 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.355+0000 7f32c0ff9700 1 -- 172.21.15.17:0/3003436208 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f32bc0d3020 con 0x7f32c41011e0 2023-07-20T02:39:27.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:26 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.17:6834/2807536449,v1:172.21.15.17:6835/2807536449] up:boot 2023-07-20T02:39:27.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:26 smithi151 ceph-mon[129163]: fsmap 3 up:standby 2023-07-20T02:39:27.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:26 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T02:39:27.477 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:39:27 smithi017 podman[161001]: 2023-07-20T02:39:27.477 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:39:27 smithi017 bash[161001]: 418b3def7e2a8a45060091f4dd1c61063e2e9a0c751fc6ff79b425640b9adc8c 2023-07-20T02:39:27.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.609+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32c402cc00 con 0x7f32c41011e0 2023-07-20T02:39:27.612 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.610+0000 7f32c0ff9700 1 -- 172.21.15.17:0/3003436208 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7f32bc0978d0 con 0x7f32c41011e0 2023-07-20T02:39:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:27.613 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T02:39:27.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f32b00a11a0 msgr2=0x7f32b00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f32b00a11a0 0x7f32b00a3660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f32c41006a0 tx=0x7f32ac00b410 comp rx=0 tx=0).stop 2023-07-20T02:39:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 msgr2=0x7f32c419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:27.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c419c980 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f32bc007a00 tx=0x7f32bc0094c0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 shutdown_connections 2023-07-20T02:39:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f32b00a11a0 0x7f32b00a3660 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f32c41011e0 0x7f32c419c980 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32c40ff650 0x7f32c41953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 --2- 172.21.15.17:0/3003436208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c4100840 0x7f32c4195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.613+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 >> 172.21.15.17:0/3003436208 conn(0x7f32c40faba0 msgr2=0x7f32c40fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:27.618 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.614+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 shutdown_connections 2023-07-20T02:39:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:27.614+0000 7f32c9fb1700 1 -- 172.21.15.17:0/3003436208 wait complete. 2023-07-20T02:39:27.619 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 4 2023-07-20T02:39:27.954 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:39:27 smithi017 ceph-mds[161084]: -- 172.21.15.17:0/4250327630 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56013d02b1e0 con 0x56013d000800 2023-07-20T02:39:27.955 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:39:27 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-d[161040]: starting mds.d at 2023-07-20T02:39:27.955 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:39:27 smithi017 systemd[1]: Started Ceph mds.d for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:28.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3003436208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: Deploying daemon mds.e on smithi146 2023-07-20T02:39:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.17:6836/4223074148,v1:172.21.15.17:6837/4223074148] up:boot 2023-07-20T02:39:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: fsmap 4 up:standby 2023-07-20T02:39:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:28 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T02:39:28.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:28.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:28.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:28.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3003436208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:28.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:28.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: Deploying daemon mds.e on smithi146 2023-07-20T02:39:28.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.17:6836/4223074148,v1:172.21.15.17:6837/4223074148] up:boot 2023-07-20T02:39:28.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: fsmap 4 up:standby 2023-07-20T02:39:28.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:28 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T02:39:28.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:28.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:28.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:28.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:28.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3003436208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:28.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:28.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: Deploying daemon mds.e on smithi146 2023-07-20T02:39:28.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.17:6836/4223074148,v1:172.21.15.17:6837/4223074148] up:boot 2023-07-20T02:39:28.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: fsmap 4 up:standby 2023-07-20T02:39:28.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:28 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T02:39:29.569 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:39:29 smithi146 systemd[1]: Starting Ceph mds.e for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:29.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:29.771 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 -- 172.21.15.17:0/2962899863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa01011d0 msgr2=0x7f5fa01035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:29.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/2962899863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa01011d0 0x7f5fa01035c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009990 tx=0x7f5f90009ca0 comp rx=0 tx=0).stop 2023-07-20T02:39:29.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 -- 172.21.15.17:0/2962899863 shutdown_connections 2023-07-20T02:39:29.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/2962899863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fa0106430 0x7f5fa01088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:29.772 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/2962899863 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa01011d0 0x7f5fa01035c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:29.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/2962899863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa0103b00 0x7f5fa0105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:29.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 -- 172.21.15.17:0/2962899863 >> 172.21.15.17:0/2962899863 conn(0x7f5fa00faba0 msgr2=0x7f5fa00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:29.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 -- 172.21.15.17:0/2962899863 shutdown_connections 2023-07-20T02:39:29.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.769+0000 7f5fa4a40700 1 -- 172.21.15.17:0/2962899863 wait complete. 2023-07-20T02:39:29.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 Processor -- start 2023-07-20T02:39:29.773 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 -- start start 2023-07-20T02:39:29.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fa01011d0 0x7f5fa01963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:29.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa0103b00 0x7f5fa0196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:29.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa0106430 0x7f5fa019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:29.775 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa019df70 con 0x7f5fa0106430 2023-07-20T02:39:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa019e0b0 con 0x7f5fa0103b00 2023-07-20T02:39:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5fa4a40700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa019e240 con 0x7f5fa01011d0 2023-07-20T02:39:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5f9ed9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa0106430 0x7f5fa019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5f9ed9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa0106430 0x7f5fa019d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43384/0 (socket says 172.21.15.17:43384) 2023-07-20T02:39:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5f9ed9d700 1 -- 172.21.15.17:0/1269836124 learned_addr learned my addr 172.21.15.17:0/1269836124 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5f9dd9b700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa0103b00 0x7f5fa0196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:29.776 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.770+0000 7f5f9e59c700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fa01011d0 0x7f5fa01963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f9ed9d700 1 -- 172.21.15.17:0/1269836124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fa01011d0 msgr2=0x7f5fa01963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f9ed9d700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fa01011d0 0x7f5fa01963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f9ed9d700 1 -- 172.21.15.17:0/1269836124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa0103b00 msgr2=0x7f5fa0196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f9ed9d700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa0103b00 0x7f5fa0196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f9ed9d700 1 -- 172.21.15.17:0/1269836124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f900095f0 con 0x7f5fa0106430 2023-07-20T02:39:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f9ed9d700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa0106430 0x7f5fa019d9a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f98007ee0 tx=0x7f5f9800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f8f7fe700 1 -- 172.21.15.17:0/1269836124 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98017070 con 0x7f5fa0106430 2023-07-20T02:39:29.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f8f7fe700 1 -- 172.21.15.17:0/1269836124 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f9800f040 con 0x7f5fa0106430 2023-07-20T02:39:29.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5f8f7fe700 1 -- 172.21.15.17:0/1269836124 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98005580 con 0x7f5fa0106430 2023-07-20T02:39:29.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa019e520 con 0x7f5fa0106430 2023-07-20T02:39:29.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.771+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa019e9e0 con 0x7f5fa0106430 2023-07-20T02:39:29.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.772+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa00fc720 con 0x7f5fa0106430 2023-07-20T02:39:29.778 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.773+0000 7f5f8f7fe700 1 -- 172.21.15.17:0/1269836124 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5f9800db20 con 0x7f5fa0106430 2023-07-20T02:39:29.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.774+0000 7f5f8f7fe700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f880a1220 0x7f5f880a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:29.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.774+0000 7f5f9e59c700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f880a1220 0x7f5f880a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:29.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.774+0000 7f5f8f7fe700 1 -- 172.21.15.17:0/1269836124 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5f980cc1b0 con 0x7f5fa0106430 2023-07-20T02:39:29.779 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.775+0000 7f5f9e59c700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f880a1220 0x7f5f880a36e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000b5c0 tx=0x7f5f90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:29.780 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:29.778+0000 7f5f8f7fe700 1 -- 172.21.15.17:0/1269836124 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5f98095da0 con 0x7f5fa0106430 2023-07-20T02:39:29.909 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:39:29 smithi146 podman[153456]: 2023-07-20T02:39:30.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.026+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fa002c970 con 0x7f5fa0106430 2023-07-20T02:39:30.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.026+0000 7f5f8f7fe700 1 -- 172.21.15.17:0/1269836124 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7f5f980993c0 con 0x7f5fa0106430 2023-07-20T02:39:30.028 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:30.028 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T02:39:30.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f880a1220 msgr2=0x7f5f880a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:30.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f880a1220 0x7f5f880a36e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000b5c0 tx=0x7f5f90005fd0 comp rx=0 tx=0).stop 2023-07-20T02:39:30.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa0106430 msgr2=0x7f5fa019d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:30.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa0106430 0x7f5fa019d9a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f98007ee0 tx=0x7f5f9800d3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:30.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 shutdown_connections 2023-07-20T02:39:30.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f5f880a1220 0x7f5f880a36e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:30.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f5fa0106430 0x7f5fa019d9a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:30.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fa01011d0 0x7f5fa01963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:30.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 --2- 172.21.15.17:0/1269836124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa0103b00 0x7f5fa0196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:30.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.028+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 >> 172.21.15.17:0/1269836124 conn(0x7f5fa00faba0 msgr2=0x7f5fa0101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:30.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.029+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 shutdown_connections 2023-07-20T02:39:30.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:30.029+0000 7f5fa4a40700 1 -- 172.21.15.17:0/1269836124 wait complete. 2023-07-20T02:39:30.032 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 5 2023-07-20T02:39:30.319 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:39:29 smithi146 bash[153456]: 64aa2152cfab5a2df4c7ea4fa7a837d7c35a2b17294b296c47ca52b46c7f974f 2023-07-20T02:39:30.319 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:39:29 smithi146 ceph-mds[153496]: -- 172.21.15.146:0/3970103803 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557789e231e0 con 0x557789df8c00 2023-07-20T02:39:30.319 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:39:29 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-e[153472]: starting mds.e at 2023-07-20T02:39:30.319 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:39:29 smithi146 systemd[1]: Started Ceph mds.e for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:30.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:30.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:30.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:30.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1269836124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:30.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:30.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:30.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:30.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:30 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:30.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:30.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:30.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:30.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1269836124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:30.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:30.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:30.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:30.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:30 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:31.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1269836124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:31.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:31.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:30 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:31.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:31 smithi151 ceph-mon[129163]: Deploying daemon mds.f on smithi151 2023-07-20T02:39:31.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:31 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.146:6834/2557911660,v1:172.21.15.146:6835/2557911660] up:boot 2023-07-20T02:39:31.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:31 smithi151 ceph-mon[129163]: fsmap 5 up:standby 2023-07-20T02:39:31.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:31 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T02:39:31.864 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:39:31 smithi151 systemd[1]: Starting Ceph mds.f for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:31.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:31 smithi017 ceph-mon[127345]: Deploying daemon mds.f on smithi151 2023-07-20T02:39:31.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:31 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.146:6834/2557911660,v1:172.21.15.146:6835/2557911660] up:boot 2023-07-20T02:39:31.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:31 smithi017 ceph-mon[127345]: fsmap 5 up:standby 2023-07-20T02:39:31.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:31 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T02:39:32.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:32.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:31 smithi146 ceph-mon[128841]: Deploying daemon mds.f on smithi151 2023-07-20T02:39:32.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:31 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.146:6834/2557911660,v1:172.21.15.146:6835/2557911660] up:boot 2023-07-20T02:39:32.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:31 smithi146 ceph-mon[128841]: fsmap 5 up:standby 2023-07-20T02:39:32.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:31 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T02:39:32.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 -- 172.21.15.17:0/1804752188 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb80ff650 msgr2=0x7fcfb80ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:32.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 --2- 172.21.15.17:0/1804752188 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb80ff650 0x7fcfb80ffa70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40099c0 tx=0x7fcfb4009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 -- 172.21.15.17:0/1804752188 shutdown_connections 2023-07-20T02:39:32.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 --2- 172.21.15.17:0/1804752188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb81011e0 0x7fcfb8107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 --2- 172.21.15.17:0/1804752188 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb80ff650 0x7fcfb80ffa70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 --2- 172.21.15.17:0/1804752188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb8100840 0x7fcfb8100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 -- 172.21.15.17:0/1804752188 >> 172.21.15.17:0/1804752188 conn(0x7fcfb80faba0 msgr2=0x7fcfb80fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:32.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 -- 172.21.15.17:0/1804752188 shutdown_connections 2023-07-20T02:39:32.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 -- 172.21.15.17:0/1804752188 wait complete. 2023-07-20T02:39:32.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 Processor -- start 2023-07-20T02:39:32.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 -- start start 2023-07-20T02:39:32.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb80ff650 0x7fcfb8068460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:32.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.183+0000 7fcfc0313700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb8100840 0x7fcfb819b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:32.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfc0313700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb81011e0 0x7fcfb819d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfc0313700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb819df70 con 0x7fcfb81011e0 2023-07-20T02:39:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfc0313700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb819e0e0 con 0x7fcfb80ff650 2023-07-20T02:39:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfc0313700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb819e2a0 con 0x7fcfb8100840 2023-07-20T02:39:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb81011e0 0x7fcfb819d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb81011e0 0x7fcfb819d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43396/0 (socket says 172.21.15.17:43396) 2023-07-20T02:39:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 -- 172.21.15.17:0/2052106471 learned_addr learned my addr 172.21.15.17:0/2052106471 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbd8ae700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb8100840 0x7fcfb819b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe0af700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb80ff650 0x7fcfb8068460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 -- 172.21.15.17:0/2052106471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb8100840 msgr2=0x7fcfb819b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb8100840 0x7fcfb819b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 -- 172.21.15.17:0/2052106471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb80ff650 msgr2=0x7fcfb8068460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb80ff650 0x7fcfb8068460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 -- 172.21.15.17:0/2052106471 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb40095f0 con 0x7fcfb81011e0 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfbe8b0700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb81011e0 0x7fcfb819d7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000ef30 tx=0x7fcfb000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:32.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfaf7fe700 1 -- 172.21.15.17:0/2052106471 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0018070 con 0x7fcfb81011e0 2023-07-20T02:39:32.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfaf7fe700 1 -- 172.21.15.17:0/2052106471 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfb000f040 con 0x7fcfb81011e0 2023-07-20T02:39:32.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb819e520 con 0x7fcfb81011e0 2023-07-20T02:39:32.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfaf7fe700 1 -- 172.21.15.17:0/2052106471 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb000cdb0 con 0x7fcfb81011e0 2023-07-20T02:39:32.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.184+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb819ea30 con 0x7fcfb81011e0 2023-07-20T02:39:32.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.185+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb810ae70 con 0x7fcfb81011e0 2023-07-20T02:39:32.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.186+0000 7fcfaf7fe700 1 -- 172.21.15.17:0/2052106471 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcfb0003a00 con 0x7fcfb81011e0 2023-07-20T02:39:32.198 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.187+0000 7fcfaf7fe700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa40a1220 0x7fcfa40a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:32.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.187+0000 7fcfbe0af700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa40a1220 0x7fcfa40a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:32.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.187+0000 7fcfaf7fe700 1 -- 172.21.15.17:0/2052106471 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcfb00cca50 con 0x7fcfb81011e0 2023-07-20T02:39:32.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.188+0000 7fcfbe0af700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa40a1220 0x7fcfa40a36e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009990 tx=0x7fcfb401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:32.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.191+0000 7fcfaf7fe700 1 -- 172.21.15.17:0/2052106471 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcfb0096630 con 0x7fcfb81011e0 2023-07-20T02:39:32.352 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:39:32 smithi151 podman[154745]: 2023-07-20T02:39:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.435+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfb8106550 con 0x7fcfb81011e0 2023-07-20T02:39:32.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.436+0000 7fcfaf7fe700 1 -- 172.21.15.17:0/2052106471 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4294 (secure 0 0 0) 0x7fcfb0099c50 con 0x7fcfb81011e0 2023-07-20T02:39:32.438 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:32.438 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2557911660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2557911660},{"type":"v1","addr":"172.21.15.146:6835","nonce":2557911660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T02:39:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa40a1220 msgr2=0x7fcfa40a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa40a1220 0x7fcfa40a36e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009990 tx=0x7fcfb401a040 comp rx=0 tx=0).stop 2023-07-20T02:39:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb81011e0 msgr2=0x7fcfb819d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb81011e0 0x7fcfb819d7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000ef30 tx=0x7fcfb000c3a0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 shutdown_connections 2023-07-20T02:39:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fcfa40a1220 0x7fcfa40a36e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fcfb81011e0 0x7fcfb819d7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfb80ff650 0x7fcfb8068460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 --2- 172.21.15.17:0/2052106471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfb8100840 0x7fcfb819b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 >> 172.21.15.17:0/2052106471 conn(0x7fcfb80faba0 msgr2=0x7fcfb80fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 shutdown_connections 2023-07-20T02:39:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:32.438+0000 7fcfc0313700 1 -- 172.21.15.17:0/2052106471 wait complete. 2023-07-20T02:39:32.442 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 6 2023-07-20T02:39:32.612 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mds[154785]: -- 172.21.15.151:0/4291128889 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a91160f1e0 con 0x55a9115e5000 2023-07-20T02:39:32.613 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-f[154761]: starting mds.f at 2023-07-20T02:39:32.613 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:39:32 smithi151 bash[154745]: cbce14c07031b1a6bd5f670b125c13c22abbb3252188ce981deb2c2761bf9236 2023-07-20T02:39:32.613 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:39:32 smithi151 systemd[1]: Started Ceph mds.f for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:32.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:32.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2052106471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:32.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:32.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:32.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:32.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:32.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:32.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:32 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:32.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:32.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2052106471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:32 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:33.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:33.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2052106471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:33.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:33.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:33.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:33.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:33.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:33.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:32 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:33.710 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:33 smithi151 ceph-mon[129163]: Deploying daemon mds.c on smithi151 2023-07-20T02:39:33.711 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:33 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.151:6834/1044630552,v1:172.21.15.151:6835/1044630552] up:boot 2023-07-20T02:39:33.711 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:33 smithi151 ceph-mon[129163]: fsmap 6 up:standby 2023-07-20T02:39:33.711 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:33 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T02:39:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:33 smithi017 ceph-mon[127345]: Deploying daemon mds.c on smithi151 2023-07-20T02:39:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:33 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.151:6834/1044630552,v1:172.21.15.151:6835/1044630552] up:boot 2023-07-20T02:39:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:33 smithi017 ceph-mon[127345]: fsmap 6 up:standby 2023-07-20T02:39:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:33 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T02:39:33.960 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:39:33 smithi151 systemd[1]: Starting Ceph mds.c for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:34.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:33 smithi146 ceph-mon[128841]: Deploying daemon mds.c on smithi151 2023-07-20T02:39:34.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:33 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.151:6834/1044630552,v1:172.21.15.151:6835/1044630552] up:boot 2023-07-20T02:39:34.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:33 smithi146 ceph-mon[128841]: fsmap 6 up:standby 2023-07-20T02:39:34.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:33 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T02:39:34.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:34.544 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:39:34 smithi151 podman[155085]: 2023-07-20T02:39:34.595 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 -- 172.21.15.17:0/3639375663 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 msgr2=0x7fb348100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:34.595 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 --2- 172.21.15.17:0/3639375663 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348100b90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3380099c0 tx=0x7fb338009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.595 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 -- 172.21.15.17:0/3639375663 shutdown_connections 2023-07-20T02:39:34.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 --2- 172.21.15.17:0/3639375663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb348102300 0x7fb348108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 --2- 172.21.15.17:0/3639375663 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348100b90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 --2- 172.21.15.17:0/3639375663 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb348101960 0x7fb348101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 -- 172.21.15.17:0/3639375663 >> 172.21.15.17:0/3639375663 conn(0x7fb3480fbcc0 msgr2=0x7fb3480fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:34.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 -- 172.21.15.17:0/3639375663 shutdown_connections 2023-07-20T02:39:34.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 -- 172.21.15.17:0/3639375663 wait complete. 2023-07-20T02:39:34.596 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.593+0000 7fb34f32d700 1 Processor -- start 2023-07-20T02:39:34.597 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34f32d700 1 -- start start 2023-07-20T02:39:34.598 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34f32d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34f32d700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb348101960 0x7fb348072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34f32d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb348102300 0x7fb3480730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34f32d700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb348073840 con 0x7fb348100770 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34f32d700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480739b0 con 0x7fb348101960 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34f32d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3481a03d0 con 0x7fb348102300 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43406/0 (socket says 172.21.15.17:43406) 2023-07-20T02:39:34.599 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 -- 172.21.15.17:0/350172336 learned_addr learned my addr 172.21.15.17:0/350172336 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:34.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34c8c8700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb348101960 0x7fb348072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:34.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d8ca700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb348102300 0x7fb3480730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:34.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 -- 172.21.15.17:0/350172336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb348102300 msgr2=0x7fb3480730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:34.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb348102300 0x7fb3480730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 -- 172.21.15.17:0/350172336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb348101960 msgr2=0x7fb348072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:34.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb348101960 0x7fb348072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.600 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 -- 172.21.15.17:0/350172336 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3380095f0 con 0x7fb348100770 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d8ca700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb348102300 0x7fb3480730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb34d0c9700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348074520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb338009f50 tx=0x7fb338005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.594+0000 7fb33e7fc700 1 -- 172.21.15.17:0/350172336 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33801d070 con 0x7fb348100770 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.595+0000 7fb33e7fc700 1 -- 172.21.15.17:0/350172336 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb338004010 con 0x7fb348100770 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.595+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3481a0650 con 0x7fb348100770 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.595+0000 7fb33e7fc700 1 -- 172.21.15.17:0/350172336 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338022cd0 con 0x7fb348100770 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.595+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3481a0b50 con 0x7fb348100770 2023-07-20T02:39:34.601 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.595+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb348107c70 con 0x7fb348100770 2023-07-20T02:39:34.602 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.597+0000 7fb33e7fc700 1 -- 172.21.15.17:0/350172336 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb338022470 con 0x7fb348100770 2023-07-20T02:39:34.602 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.597+0000 7fb33e7fc700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb3340a1220 0x7fb3340a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:34.602 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.597+0000 7fb33e7fc700 1 -- 172.21.15.17:0/350172336 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb3380cdeb0 con 0x7fb348100770 2023-07-20T02:39:34.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.597+0000 7fb34c8c8700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb3340a1220 0x7fb3340a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:34.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.598+0000 7fb34c8c8700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb3340a1220 0x7fb3340a36e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb34400ac50 tx=0x7fb34400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:34.603 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.601+0000 7fb33e7fc700 1 -- 172.21.15.17:0/350172336 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb338097b30 con 0x7fb348100770 2023-07-20T02:39:34.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.849+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb34802c970 con 0x7fb348100770 2023-07-20T02:39:34.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.850+0000 7fb33e7fc700 1 -- 172.21.15.17:0/350172336 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5054 (secure 0 0 0) 0x7fb33809b150 con 0x7fb348100770 2023-07-20T02:39:34.852 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:34.852 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2557911660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2557911660},{"type":"v1","addr":"172.21.15.146:6835","nonce":2557911660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/1044630552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":1044630552},{"type":"v1","addr":"172.21.15.151:6835","nonce":1044630552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-20T02:39:34.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb3340a1220 msgr2=0x7fb3340a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:34.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb3340a1220 0x7fb3340a36e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb34400ac50 tx=0x7fb34400a380 comp rx=0 tx=0).stop 2023-07-20T02:39:34.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 msgr2=0x7fb348074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:34.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348074520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb338009f50 tx=0x7fb338005720 comp rx=0 tx=0).stop 2023-07-20T02:39:34.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 shutdown_connections 2023-07-20T02:39:34.854 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb3340a1220 0x7fb3340a36e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.855 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb348102300 0x7fb3480730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.855 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb348100770 0x7fb348074520 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.855 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 --2- 172.21.15.17:0/350172336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb348101960 0x7fb348072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:34.855 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 >> 172.21.15.17:0/350172336 conn(0x7fb3480fbcc0 msgr2=0x7fb348104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:34.855 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 shutdown_connections 2023-07-20T02:39:34.855 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:34.852+0000 7fb34f32d700 1 -- 172.21.15.17:0/350172336 wait complete. 2023-07-20T02:39:34.856 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 7 2023-07-20T02:39:34.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:34 smithi151 ceph-mon[129163]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:34.863 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:39:34 smithi151 ceph-mds[155125]: -- 172.21.15.151:0/3988453395 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0ea5eb1e0 con 0x55d0ea5c0c00 2023-07-20T02:39:34.863 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:39:34 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-c[155101]: starting mds.c at 2023-07-20T02:39:34.863 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:39:34 smithi151 bash[155085]: 34879c4dfc391b8dbe09edae617990dcaf37309817b9ddbf00837bf0ca104e09 2023-07-20T02:39:34.863 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:39:34 smithi151 systemd[1]: Started Ceph mds.c for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:34.866 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:34 smithi017 ceph-mon[127345]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:35.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:34 smithi146 ceph-mon[128841]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:35.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:35.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:35.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:35.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:35.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:35.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:35.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: Deploying daemon mds.h on smithi146 2023-07-20T02:39:35.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/350172336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:35.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.151:6836/2694596546,v1:172.21.15.151:6837/2694596546] up:boot 2023-07-20T02:39:35.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: fsmap 7 up:standby 2023-07-20T02:39:35.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:35 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T02:39:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:35.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:35.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:35.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: Deploying daemon mds.h on smithi146 2023-07-20T02:39:35.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/350172336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:35.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.151:6836/2694596546,v1:172.21.15.151:6837/2694596546] up:boot 2023-07-20T02:39:35.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: fsmap 7 up:standby 2023-07-20T02:39:35.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:35 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T02:39:36.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:36.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:36.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:36.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:36.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:36.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:36.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: Deploying daemon mds.h on smithi146 2023-07-20T02:39:36.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/350172336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:36.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.151:6836/2694596546,v1:172.21.15.151:6837/2694596546] up:boot 2023-07-20T02:39:36.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: fsmap 7 up:standby 2023-07-20T02:39:36.115 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:35 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T02:39:36.666 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:39:36 smithi146 systemd[1]: Starting Ceph mds.h for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:36.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:36.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:36 smithi146 ceph-mon[128841]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:36.924 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:39:36 smithi146 podman[153795]: 2023-07-20T02:39:36.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:36 smithi017 ceph-mon[127345]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:37.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 -- 172.21.15.17:0/2850287005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd08106430 msgr2=0x7fbd081088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:37.009 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 --2- 172.21.15.17:0/2850287005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd08106430 0x7fbd081088a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000b0a0 tx=0x7fbd0000b3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 -- 172.21.15.17:0/2850287005 shutdown_connections 2023-07-20T02:39:37.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 --2- 172.21.15.17:0/2850287005 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd08106430 0x7fbd081088a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 --2- 172.21.15.17:0/2850287005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd081011d0 0x7fbd081035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 --2- 172.21.15.17:0/2850287005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd08103b00 0x7fbd08105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 -- 172.21.15.17:0/2850287005 >> 172.21.15.17:0/2850287005 conn(0x7fbd080faba0 msgr2=0x7fbd080fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:37.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 -- 172.21.15.17:0/2850287005 shutdown_connections 2023-07-20T02:39:37.010 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 -- 172.21.15.17:0/2850287005 wait complete. 2023-07-20T02:39:37.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 Processor -- start 2023-07-20T02:39:37.011 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.007+0000 7fbd0e272700 1 -- start start 2023-07-20T02:39:37.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd0e272700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd081011d0 0x7fbd08196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:37.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd0e272700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08103b00 0x7fbd08196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:37.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd0e272700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd08106430 0x7fbd0819daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:37.012 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd0e272700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0819e0c0 con 0x7fbd081011d0 2023-07-20T02:39:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd0e272700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0819e200 con 0x7fbd08103b00 2023-07-20T02:39:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd0e272700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0819e390 con 0x7fbd08106430 2023-07-20T02:39:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd081011d0 0x7fbd08196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd081011d0 0x7fbd08196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43420/0 (socket says 172.21.15.17:43420) 2023-07-20T02:39:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 -- 172.21.15.17:0/3065020639 learned_addr learned my addr 172.21.15.17:0/3065020639 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd077fe700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08103b00 0x7fbd08196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd0c80f700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd08106430 0x7fbd0819daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 -- 172.21.15.17:0/3065020639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd08106430 msgr2=0x7fbd0819daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd08106430 0x7fbd0819daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 -- 172.21.15.17:0/3065020639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08103b00 msgr2=0x7fbd08196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08103b00 0x7fbd08196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.008+0000 7fbd07fff700 1 -- 172.21.15.17:0/3065020639 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd00009d00 con 0x7fbd081011d0 2023-07-20T02:39:37.014 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.009+0000 7fbd07fff700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd081011d0 0x7fbd08196540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800d980 tx=0x7fbcf800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:37.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.011+0000 7fbd057fa700 1 -- 172.21.15.17:0/3065020639 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8004220 con 0x7fbd081011d0 2023-07-20T02:39:37.015 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.011+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0819e670 con 0x7fbd081011d0 2023-07-20T02:39:37.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.011+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0819ebd0 con 0x7fbd081011d0 2023-07-20T02:39:37.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.011+0000 7fbd057fa700 1 -- 172.21.15.17:0/3065020639 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcf8007640 con 0x7fbd081011d0 2023-07-20T02:39:37.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.011+0000 7fbd057fa700 1 -- 172.21.15.17:0/3065020639 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf800bd10 con 0x7fbd081011d0 2023-07-20T02:39:37.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.012+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf4005320 con 0x7fbd081011d0 2023-07-20T02:39:37.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.013+0000 7fbd057fa700 1 -- 172.21.15.17:0/3065020639 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbcf8004380 con 0x7fbd081011d0 2023-07-20T02:39:37.016 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.014+0000 7fbd057fa700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbcf00a11a0 0x7fbcf00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.014+0000 7fbd077fe700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbcf00a11a0 0x7fbcf00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.014+0000 7fbd057fa700 1 -- 172.21.15.17:0/3065020639 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbcf80cc2f0 con 0x7fbd081011d0 2023-07-20T02:39:37.017 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.014+0000 7fbd077fe700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbcf00a11a0 0x7fbcf00a3660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc007b60 tx=0x7fbcfc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:37.020 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.018+0000 7fbd057fa700 1 -- 172.21.15.17:0/3065020639 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbcf8095ee0 con 0x7fbd081011d0 2023-07-20T02:39:37.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:36 smithi151 ceph-mon[129163]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:37.268 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.267+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcf40059f0 con 0x7fbd081011d0 2023-07-20T02:39:37.269 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.267+0000 7fbd057fa700 1 -- 172.21.15.17:0/3065020639 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5814 (secure 0 0 0) 0x7fbcf8099500 con 0x7fbd081011d0 2023-07-20T02:39:37.269 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:37.270 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2557911660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2557911660},{"type":"v1","addr":"172.21.15.146:6835","nonce":2557911660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/1044630552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":1044630552},{"type":"v1","addr":"172.21.15.151:6835","nonce":1044630552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/2694596546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":2694596546},{"type":"v1","addr":"172.21.15.151:6837","nonce":2694596546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T02:39:37.271 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.269+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbcf00a11a0 msgr2=0x7fbcf00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:37.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.269+0000 7fbd0e272700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbcf00a11a0 0x7fbcf00a3660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc007b60 tx=0x7fbcfc0073d0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.269+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd081011d0 msgr2=0x7fbd08196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:37.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.269+0000 7fbd0e272700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd081011d0 0x7fbd08196540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800d980 tx=0x7fbcf800dc90 comp rx=0 tx=0).stop 2023-07-20T02:39:37.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 shutdown_connections 2023-07-20T02:39:37.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbcf00a11a0 0x7fbcf00a3660 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd08106430 0x7fbd0819daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fbd081011d0 0x7fbd08196540 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 --2- 172.21.15.17:0/3065020639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08103b00 0x7fbd08196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:37.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 >> 172.21.15.17:0/3065020639 conn(0x7fbd080faba0 msgr2=0x7fbd08101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:37.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 shutdown_connections 2023-07-20T02:39:37.273 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:37.270+0000 7fbd0e272700 1 -- 172.21.15.17:0/3065020639 wait complete. 2023-07-20T02:39:37.274 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 8 2023-07-20T02:39:37.319 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:39:36 smithi146 ceph-mds[153835]: -- 172.21.15.146:0/1680750868 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bef71311e0 con 0x55bef7107000 2023-07-20T02:39:37.319 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:39:36 smithi146 bash[153795]: 629236dfff278be78bcb65afbe0009fa320f92073b49ea6b92f335eb93c55cf9 2023-07-20T02:39:37.319 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:39:36 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-h[153811]: starting mds.h at 2023-07-20T02:39:37.319 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:39:36 smithi146 systemd[1]: Started Ceph mds.h for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:38.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:38.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:38.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:38.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:38.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: Deploying daemon mds.i on smithi151 2023-07-20T02:39:38.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3065020639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:38.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.146:6836/2995295414,v1:172.21.15.146:6837/2995295414] up:boot 2023-07-20T02:39:38.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: fsmap 8 up:standby 2023-07-20T02:39:38.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:38 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T02:39:38.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:38.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:38.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:38.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:38.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: Deploying daemon mds.i on smithi151 2023-07-20T02:39:38.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3065020639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:38.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.146:6836/2995295414,v1:172.21.15.146:6837/2995295414] up:boot 2023-07-20T02:39:38.327 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: fsmap 8 up:standby 2023-07-20T02:39:38.327 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:38 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T02:39:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:38.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:38.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:38.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: Deploying daemon mds.i on smithi151 2023-07-20T02:39:38.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3065020639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:38.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.146:6836/2995295414,v1:172.21.15.146:6837/2995295414] up:boot 2023-07-20T02:39:38.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: fsmap 8 up:standby 2023-07-20T02:39:38.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:38 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T02:39:38.613 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:39:38 smithi151 systemd[1]: Starting Ceph mds.i for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:39.112 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:39:38 smithi151 podman[155422]: 2023-07-20T02:39:39.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.432+0000 7f1c47961700 1 -- 172.21.15.17:0/3613462992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c400ff1c0 msgr2=0x7f1c400ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.432+0000 7f1c47961700 1 --2- 172.21.15.17:0/3613462992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c400ff1c0 0x7f1c400ff670 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1c3800b0a0 tx=0x7f1c3800b3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.432+0000 7f1c47961700 1 -- 172.21.15.17:0/3613462992 shutdown_connections 2023-07-20T02:39:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.432+0000 7f1c47961700 1 --2- 172.21.15.17:0/3613462992 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c400ff1c0 0x7f1c400ff670 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.435 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.432+0000 7f1c47961700 1 --2- 172.21.15.17:0/3613462992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c40100ca0 0x7f1c401010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.432+0000 7f1c47961700 1 --2- 172.21.15.17:0/3613462992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c400fe820 0x7f1c400fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.432+0000 7f1c47961700 1 -- 172.21.15.17:0/3613462992 >> 172.21.15.17:0/3613462992 conn(0x7f1c400fa380 msgr2=0x7f1c400fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.433+0000 7f1c47961700 1 -- 172.21.15.17:0/3613462992 shutdown_connections 2023-07-20T02:39:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.433+0000 7f1c47961700 1 -- 172.21.15.17:0/3613462992 wait complete. 2023-07-20T02:39:39.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.433+0000 7f1c47961700 1 Processor -- start 2023-07-20T02:39:39.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.433+0000 7f1c47961700 1 -- start start 2023-07-20T02:39:39.438 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.433+0000 7f1c47961700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c400fe820 0x7f1c401986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.433+0000 7f1c47961700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c400ff1c0 0x7f1c40198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c47961700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c40100ca0 0x7f1c4019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c47961700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c401a0250 con 0x7f1c40100ca0 2023-07-20T02:39:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c47961700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c401a0390 con 0x7f1c400fe820 2023-07-20T02:39:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c47961700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c401a0520 con 0x7f1c400ff1c0 2023-07-20T02:39:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c40100ca0 0x7f1c4019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c40100ca0 0x7f1c4019fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43438/0 (socket says 172.21.15.17:43438) 2023-07-20T02:39:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 -- 172.21.15.17:0/742480632 learned_addr learned my addr 172.21.15.17:0/742480632 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c456fd700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c400fe820 0x7f1c401986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c44efc700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c400ff1c0 0x7f1c40198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 -- 172.21.15.17:0/742480632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c400ff1c0 msgr2=0x7f1c40198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:39.440 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c400ff1c0 0x7f1c40198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 -- 172.21.15.17:0/742480632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c400fe820 msgr2=0x7f1c401986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c400fe820 0x7f1c401986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 -- 172.21.15.17:0/742480632 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38009d00 con 0x7f1c40100ca0 2023-07-20T02:39:39.441 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c45efe700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c40100ca0 0x7f1c4019fc80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1c380062a0 tx=0x7f1c38004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c367fc700 1 -- 172.21.15.17:0/742480632 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3800e040 con 0x7f1c40100ca0 2023-07-20T02:39:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c401a07a0 con 0x7f1c40100ca0 2023-07-20T02:39:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c367fc700 1 -- 172.21.15.17:0/742480632 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c3800cec0 con 0x7f1c40100ca0 2023-07-20T02:39:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.434+0000 7f1c367fc700 1 -- 172.21.15.17:0/742480632 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38004020 con 0x7f1c40100ca0 2023-07-20T02:39:39.442 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.435+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c401a0c20 con 0x7f1c40100ca0 2023-07-20T02:39:39.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.436+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c40067a20 con 0x7f1c40100ca0 2023-07-20T02:39:39.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.437+0000 7f1c367fc700 1 -- 172.21.15.17:0/742480632 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1c3801d690 con 0x7f1c40100ca0 2023-07-20T02:39:39.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.437+0000 7f1c367fc700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1c2c0a11a0 0x7f1c2c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:39.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.438+0000 7f1c456fd700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1c2c0a11a0 0x7f1c2c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:39.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.438+0000 7f1c367fc700 1 -- 172.21.15.17:0/742480632 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1c380cdc80 con 0x7f1c40100ca0 2023-07-20T02:39:39.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.438+0000 7f1c456fd700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1c2c0a11a0 0x7f1c2c0a3660 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1c40068b60 tx=0x7f1c3c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:39.445 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.442+0000 7f1c367fc700 1 -- 172.21.15.17:0/742480632 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1c40067a20 con 0x7f1c40100ca0 2023-07-20T02:39:39.613 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:39:39 smithi151 ceph-mds[155462]: -- 172.21.15.151:0/807170270 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555c19bf71e0 con 0x555c19bccc00 2023-07-20T02:39:39.613 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:39:39 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-i[155438]: starting mds.i at 2023-07-20T02:39:39.613 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:39:39 smithi151 bash[155422]: d8c9b40c893ca6e64ae96758ad11ec70c92ad3b957040a5c80512a2132597fe4 2023-07-20T02:39:39.613 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:39:39 smithi151 systemd[1]: Started Ceph mds.i for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:39.693 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.691+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c4010a6d0 con 0x7f1c40100ca0 2023-07-20T02:39:39.694 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.692+0000 7f1c367fc700 1 -- 172.21.15.17:0/742480632 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6574 (secure 0 0 0) 0x7f1c38097900 con 0x7f1c40100ca0 2023-07-20T02:39:39.694 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:39.694 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2557911660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2557911660},{"type":"v1","addr":"172.21.15.146:6835","nonce":2557911660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2995295414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":2995295414},{"type":"v1","addr":"172.21.15.146:6837","nonce":2995295414}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/1044630552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":1044630552},{"type":"v1","addr":"172.21.15.151:6835","nonce":1044630552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/2694596546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":2694596546},{"type":"v1","addr":"172.21.15.151:6837","nonce":2694596546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T02:39:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1c2c0a11a0 msgr2=0x7f1c2c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1c2c0a11a0 0x7f1c2c0a3660 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1c40068b60 tx=0x7f1c3c00b410 comp rx=0 tx=0).stop 2023-07-20T02:39:39.697 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c40100ca0 msgr2=0x7f1c4019fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:39.698 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c40100ca0 0x7f1c4019fc80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1c380062a0 tx=0x7f1c38004d00 comp rx=0 tx=0).stop 2023-07-20T02:39:39.699 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 shutdown_connections 2023-07-20T02:39:39.699 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f1c2c0a11a0 0x7f1c2c0a3660 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.699 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f1c40100ca0 0x7f1c4019fc80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.700 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c400fe820 0x7f1c401986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.700 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 --2- 172.21.15.17:0/742480632 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c400ff1c0 0x7f1c40198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:39.700 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 >> 172.21.15.17:0/742480632 conn(0x7f1c400fa380 msgr2=0x7f1c400fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 shutdown_connections 2023-07-20T02:39:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:39.694+0000 7f1c47961700 1 -- 172.21.15.17:0/742480632 wait complete. 2023-07-20T02:39:39.701 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 9 2023-07-20T02:39:40.554 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:40.554 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: Deploying daemon mds.g on smithi017 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/742480632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:40.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.151:6838/4007685317,v1:172.21.15.151:6839/4007685317] up:boot 2023-07-20T02:39:40.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: fsmap 9 up:standby 2023-07-20T02:39:40.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:40 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T02:39:40.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:40.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:40.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:40.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:40.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: Deploying daemon mds.g on smithi017 2023-07-20T02:39:40.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/742480632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:40.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.151:6838/4007685317,v1:172.21.15.151:6839/4007685317] up:boot 2023-07-20T02:39:40.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: fsmap 9 up:standby 2023-07-20T02:39:40.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:40 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T02:39:40.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:40.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:40.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:40.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:40.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:40.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: Deploying daemon mds.g on smithi017 2023-07-20T02:39:40.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/742480632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:40.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.151:6838/4007685317,v1:172.21.15.151:6839/4007685317] up:boot 2023-07-20T02:39:40.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: fsmap 9 up:standby 2023-07-20T02:39:40.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:40 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T02:39:40.878 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:39:40 smithi017 systemd[1]: Starting Ceph mds.g for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:41.455 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:39:41 smithi017 podman[161613]: 2023-07-20T02:39:41.455 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:39:41 smithi017 bash[161613]: 9f24b460eac9de3f4e3b3dd3a669bdbab1dad021768f433d798b04c52494bb07 2023-07-20T02:39:41.455 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:39:41 smithi017 ceph-mds[161653]: -- 172.21.15.17:0/2736068085 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ed00ea71e0 con 0x55ed00e7c800 2023-07-20T02:39:41.456 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:39:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-g[161629]: starting mds.g at 2023-07-20T02:39:41.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:41.738 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:39:41 smithi017 systemd[1]: Started Ceph mds.g for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:41.843 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 -- 172.21.15.17:0/4002334525 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd7181011e0 msgr2=0x7fd718107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4002334525 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd7181011e0 0x7fd718107a90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd71000b0d0 tx=0x7fd71000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 -- 172.21.15.17:0/4002334525 shutdown_connections 2023-07-20T02:39:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4002334525 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd7181011e0 0x7fd718107a90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4002334525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7180ff650 0x7fd7180ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4002334525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd718100840 0x7fd718100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 -- 172.21.15.17:0/4002334525 >> 172.21.15.17:0/4002334525 conn(0x7fd7180faba0 msgr2=0x7fd7180fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 -- 172.21.15.17:0/4002334525 shutdown_connections 2023-07-20T02:39:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 -- 172.21.15.17:0/4002334525 wait complete. 2023-07-20T02:39:41.845 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.841+0000 7fd71d57c700 1 Processor -- start 2023-07-20T02:39:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd71d57c700 1 -- start start 2023-07-20T02:39:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd71d57c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7180ff650 0x7fd718195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd71d57c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd718100840 0x7fd7181959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd71d57c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7181011e0 0x7fd71819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:41.848 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd71d57c700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71819cff0 con 0x7fd718100840 2023-07-20T02:39:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd71d57c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71819d130 con 0x7fd7181011e0 2023-07-20T02:39:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd71d57c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71819d2c0 con 0x7fd7180ff650 2023-07-20T02:39:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd718100840 0x7fd7181959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd718100840 0x7fd7181959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43460/0 (socket says 172.21.15.17:43460) 2023-07-20T02:39:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 -- 172.21.15.17:0/4259043612 learned_addr learned my addr 172.21.15.17:0/4259043612 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7177fe700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7181011e0 0x7fd71819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:41.849 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd716ffd700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7180ff650 0x7fd718195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 -- 172.21.15.17:0/4259043612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7180ff650 msgr2=0x7fd718195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7180ff650 0x7fd718195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 -- 172.21.15.17:0/4259043612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7181011e0 msgr2=0x7fd71819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7181011e0 0x7fd71819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 -- 172.21.15.17:0/4259043612 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd710009d00 con 0x7fd718100840 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd716ffd700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7180ff650 0x7fd718195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.842+0000 7fd7167fc700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd718100840 0x7fd7181959b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00d950 tx=0x7fd70c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:41.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.843+0000 7fd6fffff700 1 -- 172.21.15.17:0/4259043612 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c009940 con 0x7fd718100840 2023-07-20T02:39:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.843+0000 7fd6fffff700 1 -- 172.21.15.17:0/4259043612 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd70c010460 con 0x7fd718100840 2023-07-20T02:39:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.843+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71819d5a0 con 0x7fd718100840 2023-07-20T02:39:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.843+0000 7fd6fffff700 1 -- 172.21.15.17:0/4259043612 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00f620 con 0x7fd718100840 2023-07-20T02:39:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.843+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71819db00 con 0x7fd718100840 2023-07-20T02:39:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.844+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd718106ba0 con 0x7fd718100840 2023-07-20T02:39:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.845+0000 7fd6fffff700 1 -- 172.21.15.17:0/4259043612 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd70c0108d0 con 0x7fd718100840 2023-07-20T02:39:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.845+0000 7fd6fffff700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd7000a1220 0x7fd7000a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.845+0000 7fd716ffd700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd7000a1220 0x7fd7000a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.845+0000 7fd6fffff700 1 -- 172.21.15.17:0/4259043612 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd70c0cc1e0 con 0x7fd718100840 2023-07-20T02:39:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.846+0000 7fd716ffd700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd7000a1220 0x7fd7000a36e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd708005950 tx=0x7fd7080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:41.852 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:41.849+0000 7fd6fffff700 1 -- 172.21.15.17:0/4259043612 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd70c095dd0 con 0x7fd718100840 2023-07-20T02:39:42.108 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.106+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd71802c970 con 0x7fd718100840 2023-07-20T02:39:42.109 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.107+0000 7fd6fffff700 1 -- 172.21.15.17:0/4259043612 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7336 (secure 0 0 0) 0x7fd70c0993f0 con 0x7fd718100840 2023-07-20T02:39:42.109 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:42.110 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2557911660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2557911660},{"type":"v1","addr":"172.21.15.146:6835","nonce":2557911660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2995295414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":2995295414},{"type":"v1","addr":"172.21.15.146:6837","nonce":2995295414}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/1044630552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":1044630552},{"type":"v1","addr":"172.21.15.151:6835","nonce":1044630552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/2694596546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":2694596546},{"type":"v1","addr":"172.21.15.151:6837","nonce":2694596546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/4007685317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":4007685317},{"type":"v1","addr":"172.21.15.151:6839","nonce":4007685317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T02:39:42.111 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd7000a1220 msgr2=0x7fd7000a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:42.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd7000a1220 0x7fd7000a36e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd708005950 tx=0x7fd7080058e0 comp rx=0 tx=0).stop 2023-07-20T02:39:42.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd718100840 msgr2=0x7fd7181959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:42.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd718100840 0x7fd7181959b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00d950 tx=0x7fd70c00dc60 comp rx=0 tx=0).stop 2023-07-20T02:39:42.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 shutdown_connections 2023-07-20T02:39:42.112 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fd7000a1220 0x7fd7000a36e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:42.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7181011e0 0x7fd71819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:42.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7180ff650 0x7fd718195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:42.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 --2- 172.21.15.17:0/4259043612 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fd718100840 0x7fd7181959b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:42.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 >> 172.21.15.17:0/4259043612 conn(0x7fd7180faba0 msgr2=0x7fd718103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:42.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 shutdown_connections 2023-07-20T02:39:42.113 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:42.110+0000 7fd71d57c700 1 -- 172.21.15.17:0/4259043612 wait complete. 2023-07-20T02:39:42.114 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 10 2023-07-20T02:39:42.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:42.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: Deploying daemon mds.a on smithi017 2023-07-20T02:39:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4259043612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:42.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.17:6838/1245310151,v1:172.21.15.17:6839/1245310151] up:boot 2023-07-20T02:39:42.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: fsmap 10 up:standby 2023-07-20T02:39:42.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:42 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T02:39:42.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:42.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:42.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:42.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:42.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: Deploying daemon mds.a on smithi017 2023-07-20T02:39:42.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/4259043612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:42.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.17:6838/1245310151,v1:172.21.15.17:6839/1245310151] up:boot 2023-07-20T02:39:42.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: fsmap 10 up:standby 2023-07-20T02:39:42.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:42 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T02:39:42.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:42.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:42.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:42.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:42.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:42.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: Deploying daemon mds.a on smithi017 2023-07-20T02:39:42.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/4259043612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:42.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.17:6838/1245310151,v1:172.21.15.17:6839/1245310151] up:boot 2023-07-20T02:39:42.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: fsmap 10 up:standby 2023-07-20T02:39:42.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:42 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T02:39:43.367 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:39:43 smithi017 systemd[1]: Starting Ceph mds.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:43.668 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:39:43 smithi017 podman[161997]: 2023-07-20T02:39:43.668 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:39:43 smithi017 ceph-mds[162037]: -- 172.21.15.17:0/3232793454 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5632996a31e0 con 0x563299678800 2023-07-20T02:39:43.668 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:39:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-a[162013]: starting mds.a at 2023-07-20T02:39:43.668 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:39:43 smithi017 bash[161997]: bb1f725dcca71fd89f4109cf7cf43276df20b9dd39ceaad33f1afaa0bcc579f8 2023-07-20T02:39:43.955 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:39:43 smithi017 systemd[1]: Started Ceph mds.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:44.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:44.259 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 -- 172.21.15.17:0/1780743734 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 msgr2=0x7f2690108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 --2- 172.21.15.17:0/1780743734 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f2690108bb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f268800b0d0 tx=0x7f268800b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 -- 172.21.15.17:0/1780743734 shutdown_connections 2023-07-20T02:39:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 --2- 172.21.15.17:0/1780743734 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f2690108bb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 --2- 172.21.15.17:0/1780743734 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2690100770 0x7f2690100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 --2- 172.21.15.17:0/1780743734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2690101960 0x7f2690101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.260 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 -- 172.21.15.17:0/1780743734 >> 172.21.15.17:0/1780743734 conn(0x7f26900fbcc0 msgr2=0x7f26900fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 -- 172.21.15.17:0/1780743734 shutdown_connections 2023-07-20T02:39:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.257+0000 7f2695b36700 1 -- 172.21.15.17:0/1780743734 wait complete. 2023-07-20T02:39:44.261 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 Processor -- start 2023-07-20T02:39:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 -- start start 2023-07-20T02:39:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2690100770 0x7f2690074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2690101960 0x7f2690072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f26900730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2690073840 con 0x7f2690102300 2023-07-20T02:39:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26900739b0 con 0x7f2690100770 2023-07-20T02:39:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f2695b36700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26901a04f0 con 0x7f2690101960 2023-07-20T02:39:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f268ffff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f26900730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f268ffff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f26900730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43484/0 (socket says 172.21.15.17:43484) 2023-07-20T02:39:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f268ffff700 1 -- 172.21.15.17:0/3359089634 learned_addr learned my addr 172.21.15.17:0/3359089634 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f268effd700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2690101960 0x7f2690072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.258+0000 7f268f7fe700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2690100770 0x7f2690074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268ffff700 1 -- 172.21.15.17:0/3359089634 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2690101960 msgr2=0x7f2690072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268ffff700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2690101960 0x7f2690072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268ffff700 1 -- 172.21.15.17:0/3359089634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2690100770 msgr2=0x7f2690074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268ffff700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2690100770 0x7f2690074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268ffff700 1 -- 172.21.15.17:0/3359089634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688009d00 con 0x7f2690102300 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268effd700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2690101960 0x7f2690072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268ffff700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f26900730a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2688007990 tx=0x7f2688008f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268cff9700 1 -- 172.21.15.17:0/3359089634 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268800e040 con 0x7f2690102300 2023-07-20T02:39:44.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268cff9700 1 -- 172.21.15.17:0/3359089634 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2688005020 con 0x7f2690102300 2023-07-20T02:39:44.266 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26901a0770 con 0x7f2690102300 2023-07-20T02:39:44.266 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f268cff9700 1 -- 172.21.15.17:0/3359089634 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268801d6d0 con 0x7f2690102300 2023-07-20T02:39:44.266 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.259+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26901a0bf0 con 0x7f2690102300 2023-07-20T02:39:44.266 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.260+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2690107be0 con 0x7f2690102300 2023-07-20T02:39:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.261+0000 7f268cff9700 1 -- 172.21.15.17:0/3359089634 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2688005490 con 0x7f2690102300 2023-07-20T02:39:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.262+0000 7f268cff9700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f267c0a11a0 0x7f267c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.262+0000 7f268cff9700 1 -- 172.21.15.17:0/3359089634 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f26880cdc00 con 0x7f2690102300 2023-07-20T02:39:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.262+0000 7f268f7fe700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f267c0a11a0 0x7f267c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:44.267 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.262+0000 7f268f7fe700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f267c0a11a0 0x7f267c0a3660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2678005fd0 tx=0x7f2678005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:44.268 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.265+0000 7f268cff9700 1 -- 172.21.15.17:0/3359089634 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f26880977e0 con 0x7f2690102300 2023-07-20T02:39:44.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.511+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f269002c970 con 0x7f2690102300 2023-07-20T02:39:44.513 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.512+0000 7f268cff9700 1 -- 172.21.15.17:0/3359089634 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8094 (secure 0 0 0) 0x7f268809ae00 con 0x7f2690102300 2023-07-20T02:39:44.514 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:44.514 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6839/1245310151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6838","nonce":1245310151},{"type":"v1","addr":"172.21.15.17:6839","nonce":1245310151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2557911660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2557911660},{"type":"v1","addr":"172.21.15.146:6835","nonce":2557911660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2995295414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":2995295414},{"type":"v1","addr":"172.21.15.146:6837","nonce":2995295414}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/1044630552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":1044630552},{"type":"v1","addr":"172.21.15.151:6835","nonce":1044630552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/2694596546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":2694596546},{"type":"v1","addr":"172.21.15.151:6837","nonce":2694596546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/4007685317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":4007685317},{"type":"v1","addr":"172.21.15.151:6839","nonce":4007685317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T02:39:44.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f267c0a11a0 msgr2=0x7f267c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:44.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f267c0a11a0 0x7f267c0a3660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2678005fd0 tx=0x7f2678005e50 comp rx=0 tx=0).stop 2023-07-20T02:39:44.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 msgr2=0x7f26900730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:44.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f26900730a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2688007990 tx=0x7f2688008f60 comp rx=0 tx=0).stop 2023-07-20T02:39:44.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 shutdown_connections 2023-07-20T02:39:44.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f267c0a11a0 0x7f267c0a3660 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.517 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f2690102300 0x7f26900730a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.518 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2690100770 0x7f2690074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.518 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 --2- 172.21.15.17:0/3359089634 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2690101960 0x7f2690072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:44.518 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 >> 172.21.15.17:0/3359089634 conn(0x7f26900fbcc0 msgr2=0x7f2690104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:44.518 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 shutdown_connections 2023-07-20T02:39:44.518 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:44.514+0000 7f2695b36700 1 -- 172.21.15.17:0/3359089634 wait complete. 2023-07-20T02:39:44.518 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 11 2023-07-20T02:39:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: Deploying daemon mds.b on smithi146 2023-07-20T02:39:44.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3359089634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:44.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.17:6840/1907939772,v1:172.21.15.17:6841/1907939772] up:boot 2023-07-20T02:39:44.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: fsmap 11 up:standby 2023-07-20T02:39:44.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:44 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T02:39:45.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:45.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:45.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:45.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:45.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:45.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:45.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:45.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: Deploying daemon mds.b on smithi146 2023-07-20T02:39:45.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3359089634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:45.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.17:6840/1907939772,v1:172.21.15.17:6841/1907939772] up:boot 2023-07-20T02:39:45.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: fsmap 11 up:standby 2023-07-20T02:39:45.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:44 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T02:39:45.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:45.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:45.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:45.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: Deploying daemon mds.b on smithi146 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3359089634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.17:6840/1907939772,v1:172.21.15.17:6841/1907939772] up:boot 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: fsmap 11 up:standby 2023-07-20T02:39:45.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:44 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T02:39:45.805 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:39:45 smithi146 systemd[1]: Starting Ceph mds.b for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:46.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:45 smithi146 ceph-mon[128841]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:46.070 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:39:45 smithi146 podman[154137]: 2023-07-20T02:39:46.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:45 smithi017 ceph-mon[127345]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:46.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:45 smithi151 ceph-mon[129163]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:46.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:39:46.569 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:39:46 smithi146 ceph-mds[154177]: -- 172.21.15.146:0/120242401 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fd4066f1e0 con 0x55fd40644c00 2023-07-20T02:39:46.569 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:39:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-b[154153]: starting mds.b at 2023-07-20T02:39:46.570 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:39:46 smithi146 bash[154137]: 55daf18220a07ed3cc28b536e89f4313d34ffbe45d2928bbd49d44c44a195298 2023-07-20T02:39:46.570 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:39:46 smithi146 systemd[1]: Started Ceph mds.b for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:39:46.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- 172.21.15.17:0/1170953600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d0106430 msgr2=0x7f04d01088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:46.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 --2- 172.21.15.17:0/1170953600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d0106430 0x7f04d01088a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f04c800b0a0 tx=0x7f04c800b3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- 172.21.15.17:0/1170953600 shutdown_connections 2023-07-20T02:39:46.662 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 --2- 172.21.15.17:0/1170953600 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d0106430 0x7f04d01088a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 --2- 172.21.15.17:0/1170953600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d01011d0 0x7f04d01035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 --2- 172.21.15.17:0/1170953600 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04d0103b00 0x7f04d0105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- 172.21.15.17:0/1170953600 >> 172.21.15.17:0/1170953600 conn(0x7f04d00faba0 msgr2=0x7f04d00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- 172.21.15.17:0/1170953600 shutdown_connections 2023-07-20T02:39:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- 172.21.15.17:0/1170953600 wait complete. 2023-07-20T02:39:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 Processor -- start 2023-07-20T02:39:46.663 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- start start 2023-07-20T02:39:46.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d01011d0 0x7f04d0100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:46.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d0103b00 0x7f04d00ff180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:46.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04d0106430 0x7f04d00ff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:46.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d00ffe60 con 0x7f04d01011d0 2023-07-20T02:39:46.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.659+0000 7f04d6af2700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d00fffd0 con 0x7f04d0103b00 2023-07-20T02:39:46.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d6af2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d01a0550 con 0x7f04d0106430 2023-07-20T02:39:46.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d01011d0 0x7f04d0100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d01011d0 0x7f04d0100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43496/0 (socket says 172.21.15.17:43496) 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 -- 172.21.15.17:0/4110706341 learned_addr learned my addr 172.21.15.17:0/4110706341 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04cffff700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d0103b00 0x7f04d00ff180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d508f700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04d0106430 0x7f04d00ff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 -- 172.21.15.17:0/4110706341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04d0106430 msgr2=0x7f04d00ff6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04d0106430 0x7f04d00ff6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 -- 172.21.15.17:0/4110706341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d0103b00 msgr2=0x7f04d00ff180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d0103b00 0x7f04d00ff180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 -- 172.21.15.17:0/4110706341 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c4009520 con 0x7f04d01011d0 2023-07-20T02:39:46.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d488e700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d01011d0 0x7f04d0100b40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f04c400eb30 tx=0x7f04c400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:46.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04cdffb700 1 -- 172.21.15.17:0/4110706341 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4009880 con 0x7f04d01011d0 2023-07-20T02:39:46.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04cdffb700 1 -- 172.21.15.17:0/4110706341 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04c40044e0 con 0x7f04d01011d0 2023-07-20T02:39:46.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04cdffb700 1 -- 172.21.15.17:0/4110706341 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4003d80 con 0x7f04d01011d0 2023-07-20T02:39:46.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.660+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c8009d00 con 0x7f04d01011d0 2023-07-20T02:39:46.667 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.661+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d01a0bb0 con 0x7f04d01011d0 2023-07-20T02:39:46.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.661+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d00fc720 con 0x7f04d01011d0 2023-07-20T02:39:46.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.663+0000 7f04cdffb700 1 -- 172.21.15.17:0/4110706341 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f04c4003f00 con 0x7f04d01011d0 2023-07-20T02:39:46.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.663+0000 7f04cdffb700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f04c00a1220 0x7f04c00a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:46.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.663+0000 7f04cdffb700 1 -- 172.21.15.17:0/4110706341 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f04c4014070 con 0x7f04d01011d0 2023-07-20T02:39:46.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.663+0000 7f04cffff700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f04c00a1220 0x7f04c00a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:46.668 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.664+0000 7f04cffff700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f04c00a1220 0x7f04c00a36e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009c60 tx=0x7f04bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:46.672 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.670+0000 7f04cdffb700 1 -- 172.21.15.17:0/4110706341 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f04c4096670 con 0x7f04d01011d0 2023-07-20T02:39:46.925 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.923+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04d002ce00 con 0x7f04d01011d0 2023-07-20T02:39:46.926 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.924+0000 7f04cdffb700 1 -- 172.21.15.17:0/4110706341 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9613 (secure 0 0 0) 0x7f04c4099c90 con 0x7f04d01011d0 2023-07-20T02:39:46.926 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:46.927 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6835/2807536449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6834","nonce":2807536449},{"type":"v1","addr":"172.21.15.17:6835","nonce":2807536449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6837/4223074148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6836","nonce":4223074148},{"type":"v1","addr":"172.21.15.17:6837","nonce":4223074148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6839/1245310151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6838","nonce":1245310151},{"type":"v1","addr":"172.21.15.17:6839","nonce":1245310151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.17:6841/1907939772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6840","nonce":1907939772},{"type":"v1","addr":"172.21.15.17:6841","nonce":1907939772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1472683577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1472683577},{"type":"v1","addr":"172.21.15.146:6833","nonce":1472683577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2557911660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2557911660},{"type":"v1","addr":"172.21.15.146:6835","nonce":2557911660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/3119401116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3119401116},{"type":"v1","addr":"172.21.15.151:6833","nonce":3119401116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2995295414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":2995295414},{"type":"v1","addr":"172.21.15.146:6837","nonce":2995295414}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/1044630552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":1044630552},{"type":"v1","addr":"172.21.15.151:6835","nonce":1044630552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/2694596546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":2694596546},{"type":"v1","addr":"172.21.15.151:6837","nonce":2694596546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6839/1321727767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":1321727767},{"type":"v1","addr":"172.21.15.146:6839","nonce":1321727767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/4007685317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":4007685317},{"type":"v1","addr":"172.21.15.151:6839","nonce":4007685317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T02:39:46.928 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f04c00a1220 msgr2=0x7f04c00a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:46.928 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f04c00a1220 0x7f04c00a36e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009c60 tx=0x7f04bc009260 comp rx=0 tx=0).stop 2023-07-20T02:39:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d01011d0 msgr2=0x7f04d0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d01011d0 0x7f04d0100b40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f04c400eb30 tx=0x7f04c400ee40 comp rx=0 tx=0).stop 2023-07-20T02:39:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 shutdown_connections 2023-07-20T02:39:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f04c00a1220 0x7f04c00a36e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04d0106430 0x7f04d00ff6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f04d01011d0 0x7f04d0100b40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 --2- 172.21.15.17:0/4110706341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d0103b00 0x7f04d00ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:46.930 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 >> 172.21.15.17:0/4110706341 conn(0x7f04d00faba0 msgr2=0x7f04d0101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:46.930 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 shutdown_connections 2023-07-20T02:39:46.930 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:46.926+0000 7f04d6af2700 1 -- 172.21.15.17:0/4110706341 wait complete. 2023-07-20T02:39:46.930 INFO:teuthology.orchestra.run.smithi017.stderr:dumped fsmap epoch 13 2023-07-20T02:39:46.941 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:39:47.070 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 -- 172.21.15.17:0/2572088231 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 msgr2=0x7f92580ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.135 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2572088231 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92580ffa70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f92400099c0 tx=0x7f9240009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.135 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 -- 172.21.15.17:0/2572088231 shutdown_connections 2023-07-20T02:39:47.135 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2572088231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92581011e0 0x7f9258107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.136 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2572088231 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92580ffa70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.136 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2572088231 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9258100840 0x7f9258100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.137 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 -- 172.21.15.17:0/2572088231 >> 172.21.15.17:0/2572088231 conn(0x7f92580faba0 msgr2=0x7f92580fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:47.137 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 -- 172.21.15.17:0/2572088231 shutdown_connections 2023-07-20T02:39:47.137 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.068+0000 7f925e2ba700 1 -- 172.21.15.17:0/2572088231 wait complete. 2023-07-20T02:39:47.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 Processor -- start 2023-07-20T02:39:47.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 -- start start 2023-07-20T02:39:47.139 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92581953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.140 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9258100840 0x7f92581958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.140 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92581011e0 0x7f925819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.140 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925819cf20 con 0x7f92580ff650 2023-07-20T02:39:47.140 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92580678d0 con 0x7f92581011e0 2023-07-20T02:39:47.140 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f925e2ba700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9258067a60 con 0x7f9258100840 2023-07-20T02:39:47.140 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.069+0000 7f9257fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92581953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.140 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92581953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43510/0 (socket says 172.21.15.17:43510) 2023-07-20T02:39:47.141 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 -- 172.21.15.17:0/2597533508 learned_addr learned my addr 172.21.15.17:0/2597533508 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:47.141 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f92577fe700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9258100840 0x7f92581958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.141 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f925c857700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92581011e0 0x7f925819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.141 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 -- 172.21.15.17:0/2597533508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9258100840 msgr2=0x7f92581958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.141 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9258100840 0x7f92581958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.141 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 -- 172.21.15.17:0/2597533508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92581011e0 msgr2=0x7f925819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.142 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92581011e0 0x7f925819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.142 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 -- 172.21.15.17:0/2597533508 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92400095f0 con 0x7f92580ff650 2023-07-20T02:39:47.142 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f925c857700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92581011e0 0x7f925819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:39:47.142 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f9257fff700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92581953a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f92400036f0 tx=0x7f9240009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:47.143 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f92557fa700 1 -- 172.21.15.17:0/2597533508 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924001d070 con 0x7f92580ff650 2023-07-20T02:39:47.143 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f92557fa700 1 -- 172.21.15.17:0/2597533508 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9240004010 con 0x7f92580ff650 2023-07-20T02:39:47.143 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9258067ce0 con 0x7f92580ff650 2023-07-20T02:39:47.143 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f92557fa700 1 -- 172.21.15.17:0/2597533508 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000fe70 con 0x7f92580ff650 2023-07-20T02:39:47.143 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.070+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92580681e0 con 0x7f92580ff650 2023-07-20T02:39:47.143 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.072+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92581065d0 con 0x7f92580ff650 2023-07-20T02:39:47.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.073+0000 7f92557fa700 1 -- 172.21.15.17:0/2597533508 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f924000f560 con 0x7f92580ff650 2023-07-20T02:39:47.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.073+0000 7f92557fa700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f92440a1220 0x7f92440a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.073+0000 7f92577fe700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f92440a1220 0x7f92440a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.073+0000 7f92557fa700 1 -- 172.21.15.17:0/2597533508 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92400cd320 con 0x7f92580ff650 2023-07-20T02:39:47.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.074+0000 7f92577fe700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f92440a1220 0x7f92440a36e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9248004630 tx=0x7f924800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:47.144 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.078+0000 7f92557fa700 1 -- 172.21.15.17:0/2597533508 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9240096fa0 con 0x7f92580ff650 2023-07-20T02:39:47.289 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.287+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9258068580 con 0x7f92580ff650 2023-07-20T02:39:47.291 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.289+0000 7f92557fa700 1 -- 172.21.15.17:0/2597533508 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18421 (secure 0 0 0) 0x7f924009a5c0 con 0x7f92580ff650 2023-07-20T02:39:47.292 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:47.292 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":79,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","created":"2023-07-20T02:32:50.175394+0000","modified":"2023-07-20T02:39:10.303454+0000","last_up_change":"2023-07-20T02:39:09.254013+0000","last_in_change":"2023-07-20T02:38:51.253472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:35:55.360132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb89ee2e-e4ff-4c83-9626-69cbfb526dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6803","nonce":795351069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6805","nonce":795351069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6809","nonce":795351069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6807","nonce":795351069}]},"public_addr":"172.21.15.17:6803/795351069","cluster_addr":"172.21.15.17:6805/795351069","heartbeat_back_addr":"172.21.15.17:6809/795351069","heartbeat_front_addr":"172.21.15.17:6807/795351069","state":["exists","up"]},{"osd":1,"uuid":"2d422389-0925-410c-bd28-50ad06d68cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6801","nonce":3123864696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6803","nonce":3123864696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6807","nonce":3123864696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6805","nonce":3123864696}]},"public_addr":"172.21.15.146:6801/3123864696","cluster_addr":"172.21.15.146:6803/3123864696","heartbeat_back_addr":"172.21.15.146:6807/3123864696","heartbeat_front_addr":"172.21.15.146:6805/3123864696","state":["exists","up"]},{"osd":2,"uuid":"a08d37e7-a3cd-4694-9441-b505bd167c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6801","nonce":1299864358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6803","nonce":1299864358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6807","nonce":1299864358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6805","nonce":1299864358}]},"public_addr":"172.21.15.151:6801/1299864358","cluster_addr":"172.21.15.151:6803/1299864358","heartbeat_back_addr":"172.21.15.151:6807/1299864358","heartbeat_front_addr":"172.21.15.151:6805/1299864358","state":["exists","up"]},{"osd":3,"uuid":"62ad61a1-05b0-4e38-a2eb-530485d806b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6811","nonce":604342850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6813","nonce":604342850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6817","nonce":604342850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6815","nonce":604342850}]},"public_addr":"172.21.15.17:6811/604342850","cluster_addr":"172.21.15.17:6813/604342850","heartbeat_back_addr":"172.21.15.17:6817/604342850","heartbeat_front_addr":"172.21.15.17:6815/604342850","state":["exists","up"]},{"osd":4,"uuid":"38ed288f-9c0d-415c-b52f-075d3b7599db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6809","nonce":4024595090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6811","nonce":4024595090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6815","nonce":4024595090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6813","nonce":4024595090}]},"public_addr":"172.21.15.146:6809/4024595090","cluster_addr":"172.21.15.146:6811/4024595090","heartbeat_back_addr":"172.21.15.146:6815/4024595090","heartbeat_front_addr":"172.21.15.146:6813/4024595090","state":["exists","up"]},{"osd":5,"uuid":"7da194cb-67fb-4c73-b84c-fa899c3635b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6809","nonce":1633219838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6811","nonce":1633219838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6815","nonce":1633219838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6813","nonce":1633219838}]},"public_addr":"172.21.15.151:6809/1633219838","cluster_addr":"172.21.15.151:6811/1633219838","heartbeat_back_addr":"172.21.15.151:6815/1633219838","heartbeat_front_addr":"172.21.15.151:6813/1633219838","state":["exists","up"]},{"osd":6,"uuid":"67ced483-d37a-46c1-97ae-288f28b091f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6819","nonce":3807389822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6821","nonce":3807389822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6825","nonce":3807389822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6823","nonce":3807389822}]},"public_addr":"172.21.15.17:6819/3807389822","cluster_addr":"172.21.15.17:6821/3807389822","heartbeat_back_addr":"172.21.15.17:6825/3807389822","heartbeat_front_addr":"172.21.15.17:6823/3807389822","state":["exists","up"]},{"osd":7,"uuid":"26199bbc-d82f-4e97-9218-0058426ef412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6817","nonce":708669673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6819","nonce":708669673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6823","nonce":708669673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6821","nonce":708669673}]},"public_addr":"172.21.15.146:6817/708669673","cluster_addr":"172.21.15.146:6819/708669673","heartbeat_back_addr":"172.21.15.146:6823/708669673","heartbeat_front_addr":"172.21.15.146:6821/708669673","state":["exists","up"]},{"osd":8,"uuid":"b1a33778-9ab5-45db-90ff-644794ee1cd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6817","nonce":2042667088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6819","nonce":2042667088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6823","nonce":2042667088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6821","nonce":2042667088}]},"public_addr":"172.21.15.151:6817/2042667088","cluster_addr":"172.21.15.151:6819/2042667088","heartbeat_back_addr":"172.21.15.151:6823/2042667088","heartbeat_front_addr":"172.21.15.151:6821/2042667088","state":["exists","up"]},{"osd":9,"uuid":"496d04b3-61a4-4405-aa52-5590aacf7129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6827","nonce":1403136476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6829","nonce":1403136476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6833","nonce":1403136476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6831","nonce":1403136476}]},"public_addr":"172.21.15.17:6827/1403136476","cluster_addr":"172.21.15.17:6829/1403136476","heartbeat_back_addr":"172.21.15.17:6833/1403136476","heartbeat_front_addr":"172.21.15.17:6831/1403136476","state":["exists","up"]},{"osd":10,"uuid":"6ee41a91-3ac3-4262-bff2-7531048859c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6825","nonce":2995236638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6827","nonce":2995236638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6831","nonce":2995236638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6829","nonce":2995236638}]},"public_addr":"172.21.15.146:6825/2995236638","cluster_addr":"172.21.15.146:6827/2995236638","heartbeat_back_addr":"172.21.15.146:6831/2995236638","heartbeat_front_addr":"172.21.15.146:6829/2995236638","state":["exists","up"]},{"osd":11,"uuid":"282d07a4-1bb0-4239-9c1e-adeebe9d7432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6825","nonce":1663985079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6827","nonce":1663985079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6831","nonce":1663985079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6829","nonce":1663985079}]},"public_addr":"172.21.15.151:6825/1663985079","cluster_addr":"172.21.15.151:6827/1663985079","heartbeat_back_addr":"172.21.15.151:6831/1663985079","heartbeat_front_addr":"172.21.15.151:6829/1663985079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:07.560935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:29.818964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:51.802583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:12.985262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:34.792736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:56.378329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:17.498840+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:40.421290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:02.192805+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:23.342606+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:45.695709+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:39:07.701215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2979320785":"2023-07-21T02:33:44.823257+0000","172.21.15.17:0/979357839":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6800/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/714275157":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1888486288":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/910909892":"2023-07-21T02:33:19.367142+0000","172.21.15.17:6800/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1536507180":"2023-07-21T02:33:44.823257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:39:47.294 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f92440a1220 msgr2=0x7f92440a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.294 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f92440a1220 0x7f92440a36e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9248004630 tx=0x7f924800b3c0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 msgr2=0x7f92581953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92581953a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f92400036f0 tx=0x7f9240009580 comp rx=0 tx=0).stop 2023-07-20T02:39:47.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 shutdown_connections 2023-07-20T02:39:47.295 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f92440a1220 0x7f92440a36e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.296 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92581011e0 0x7f925819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.296 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f92580ff650 0x7f92581953a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.296 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 --2- 172.21.15.17:0/2597533508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9258100840 0x7f92581958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.297 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 >> 172.21.15.17:0/2597533508 conn(0x7f92580faba0 msgr2=0x7f92580fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:47.297 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 shutdown_connections 2023-07-20T02:39:47.297 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.291+0000 7f925e2ba700 1 -- 172.21.15.17:0/2597533508 wait complete. 2023-07-20T02:39:47.306 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:35:55.360132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T02:39:47.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T02:39:47.442 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 -- 172.21.15.17:0/320646106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 msgr2=0x7fc008100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 --2- 172.21.15.17:0/320646106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc008100ce0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbff80099c0 tx=0x7fbff8009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 -- 172.21.15.17:0/320646106 shutdown_connections 2023-07-20T02:39:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 --2- 172.21.15.17:0/320646106 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc008101220 0x7fc008107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 --2- 172.21.15.17:0/320646106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0080ff690 0x7fc0080ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 --2- 172.21.15.17:0/320646106 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc008100ce0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 -- 172.21.15.17:0/320646106 >> 172.21.15.17:0/320646106 conn(0x7fc0080fac00 msgr2=0x7fc0080fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 -- 172.21.15.17:0/320646106 shutdown_connections 2023-07-20T02:39:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.440+0000 7fc010155700 1 -- 172.21.15.17:0/320646106 wait complete. 2023-07-20T02:39:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 Processor -- start 2023-07-20T02:39:47.444 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 -- start start 2023-07-20T02:39:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0080ff690 0x7fc008069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc00819b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc008101220 0x7fc00819d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00819df50 con 0x7fc008100880 2023-07-20T02:39:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00819e0c0 con 0x7fc008101220 2023-07-20T02:39:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc010155700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00819e280 con 0x7fc0080ff690 2023-07-20T02:39:47.446 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00d6f0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc00819b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00d6f0700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc00819b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43526/0 (socket says 172.21.15.17:43526) 2023-07-20T02:39:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00d6f0700 1 -- 172.21.15.17:0/3042454752 learned_addr learned my addr 172.21.15.17:0/3042454752 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00e6f2700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc008101220 0x7fc00819d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00def1700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0080ff690 0x7fc008069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00d6f0700 1 -- 172.21.15.17:0/3042454752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0080ff690 msgr2=0x7fc008069430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00d6f0700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0080ff690 0x7fc008069430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.447 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00d6f0700 1 -- 172.21.15.17:0/3042454752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc008101220 msgr2=0x7fc00819d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.441+0000 7fc00d6f0700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc008101220 0x7fc00819d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.442+0000 7fc00d6f0700 1 -- 172.21.15.17:0/3042454752 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff80095f0 con 0x7fc008100880 2023-07-20T02:39:47.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.442+0000 7fc00d6f0700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc00819b130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009990 tx=0x7fbff8003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:47.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.442+0000 7fbffeffd700 1 -- 172.21.15.17:0/3042454752 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff801d070 con 0x7fc008100880 2023-07-20T02:39:47.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.442+0000 7fbffeffd700 1 -- 172.21.15.17:0/3042454752 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbff8004640 con 0x7fc008100880 2023-07-20T02:39:47.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.442+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00819e500 con 0x7fc008100880 2023-07-20T02:39:47.448 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.442+0000 7fbffeffd700 1 -- 172.21.15.17:0/3042454752 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff8022e90 con 0x7fc008100880 2023-07-20T02:39:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.442+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00819ea00 con 0x7fc008100880 2023-07-20T02:39:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.443+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc008106b80 con 0x7fc008100880 2023-07-20T02:39:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.444+0000 7fbffeffd700 1 -- 172.21.15.17:0/3042454752 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbff8003fb0 con 0x7fc008100880 2023-07-20T02:39:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.445+0000 7fbffeffd700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbff40a11d0 0x7fbff40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.445+0000 7fc00def1700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbff40a11d0 0x7fbff40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.445+0000 7fbffeffd700 1 -- 172.21.15.17:0/3042454752 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbff80cdd70 con 0x7fc008100880 2023-07-20T02:39:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.445+0000 7fc00def1700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbff40a11d0 0x7fbff40a3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc0081006e0 tx=0x7fc00400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:47.450 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.448+0000 7fbffeffd700 1 -- 172.21.15.17:0/3042454752 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbff80979d0 con 0x7fc008100880 2023-07-20T02:39:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:47.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.146:6838/1321727767,v1:172.21.15.146:6839/1321727767] up:boot 2023-07-20T02:39:47.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: fsmap 12 up:standby 2023-07-20T02:39:47.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T02:39:47.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:47 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4110706341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:47.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:47.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.146:6838/1321727767,v1:172.21.15.146:6839/1321727767] up:boot 2023-07-20T02:39:47.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: fsmap 12 up:standby 2023-07-20T02:39:47.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T02:39:47.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:47 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/4110706341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:47.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:47.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:47.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.146:6838/1321727767,v1:172.21.15.146:6839/1321727767] up:boot 2023-07-20T02:39:47.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: fsmap 12 up:standby 2023-07-20T02:39:47.615 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T02:39:47.615 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:47 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/4110706341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:39:47.660 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.658+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc00802cbc0 con 0x7fc008100880 2023-07-20T02:39:47.661 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.659+0000 7fbffeffd700 1 -- 172.21.15.17:0/3042454752 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbff809aff0 con 0x7fc008100880 2023-07-20T02:39:47.662 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2023-07-20T02:39:47.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbff40a11d0 msgr2=0x7fbff40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbff40a11d0 0x7fbff40a3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc0081006e0 tx=0x7fc00400a3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 msgr2=0x7fc00819b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.664 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc00819b130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009990 tx=0x7fbff8003af0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 shutdown_connections 2023-07-20T02:39:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fbff40a11d0 0x7fbff40a3690 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc008101220 0x7fc00819d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0080ff690 0x7fc008069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 --2- 172.21.15.17:0/3042454752 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc008100880 0x7fc00819b130 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 >> 172.21.15.17:0/3042454752 conn(0x7fc0080fac00 msgr2=0x7fc008103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 shutdown_connections 2023-07-20T02:39:47.666 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.662+0000 7fc010155700 1 -- 172.21.15.17:0/3042454752 wait complete. 2023-07-20T02:39:47.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-20T02:39:47.806 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 -- 172.21.15.17:0/2799681065 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 msgr2=0x7f531c0ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 --2- 172.21.15.17:0/2799681065 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c0ffa80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f53100099c0 tx=0x7f5310009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 -- 172.21.15.17:0/2799681065 shutdown_connections 2023-07-20T02:39:47.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 --2- 172.21.15.17:0/2799681065 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f531c1011f0 0x7f531c107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 --2- 172.21.15.17:0/2799681065 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c0ffa80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 --2- 172.21.15.17:0/2799681065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f531c100850 0x7f531c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.807 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 -- 172.21.15.17:0/2799681065 >> 172.21.15.17:0/2799681065 conn(0x7f531c0fab90 msgr2=0x7f531c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:47.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 -- 172.21.15.17:0/2799681065 shutdown_connections 2023-07-20T02:39:47.808 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.804+0000 7f5320824700 1 -- 172.21.15.17:0/2799681065 wait complete. 2023-07-20T02:39:47.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 Processor -- start 2023-07-20T02:39:47.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 -- start start 2023-07-20T02:39:47.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f531c100850 0x7f531c19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.810 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f531c1011f0 0x7f531c19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c19df40 con 0x7f531c0ff660 2023-07-20T02:39:47.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c19e0b0 con 0x7f531c100850 2023-07-20T02:39:47.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5320824700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c19e270 con 0x7f531c1011f0 2023-07-20T02:39:47.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f531a59c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f531a59c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c069470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43538/0 (socket says 172.21.15.17:43538) 2023-07-20T02:39:47.811 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f531a59c700 1 -- 172.21.15.17:0/4023142067 learned_addr learned my addr 172.21.15.17:0/4023142067 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:47.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f5319d9b700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f531c100850 0x7f531c19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.805+0000 7f531ad9d700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f531c1011f0 0x7f531c19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f531a59c700 1 -- 172.21.15.17:0/4023142067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f531c1011f0 msgr2=0x7f531c19d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f531a59c700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f531c1011f0 0x7f531c19d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f531a59c700 1 -- 172.21.15.17:0/4023142067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f531c100850 msgr2=0x7f531c19b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:47.812 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f531a59c700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f531c100850 0x7f531c19b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:47.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f531a59c700 1 -- 172.21.15.17:0/4023142067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53100095f0 con 0x7f531c0ff660 2023-07-20T02:39:47.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f5319d9b700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f531c100850 0x7f531c19b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:39:47.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f531a59c700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c069470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5310009fd0 tx=0x7f531000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:47.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f530b7fe700 1 -- 172.21.15.17:0/4023142067 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531001d070 con 0x7f531c0ff660 2023-07-20T02:39:47.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f530b7fe700 1 -- 172.21.15.17:0/4023142067 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5310005030 con 0x7f531c0ff660 2023-07-20T02:39:47.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c19e4f0 con 0x7f531c0ff660 2023-07-20T02:39:47.813 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f530b7fe700 1 -- 172.21.15.17:0/4023142067 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310004aa0 con 0x7f531c0ff660 2023-07-20T02:39:47.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.806+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c19e910 con 0x7f531c0ff660 2023-07-20T02:39:47.814 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.807+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c10ae80 con 0x7f531c0ff660 2023-07-20T02:39:47.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.809+0000 7f530b7fe700 1 -- 172.21.15.17:0/4023142067 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5310005410 con 0x7f531c0ff660 2023-07-20T02:39:47.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.809+0000 7f530b7fe700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f53040a11a0 0x7f53040a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:47.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.810+0000 7f5319d9b700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f53040a11a0 0x7f53040a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:47.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.810+0000 7f530b7fe700 1 -- 172.21.15.17:0/4023142067 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f53100cd750 con 0x7f531c0ff660 2023-07-20T02:39:47.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.810+0000 7f5319d9b700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f53040a11a0 0x7f53040a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f530c005950 tx=0x7f530c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:47.815 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:47.813+0000 7f530b7fe700 1 -- 172.21.15.17:0/4023142067 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5310097330 con 0x7f531c0ff660 2023-07-20T02:39:48.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.033+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f531c106560 con 0x7f531c0ff660 2023-07-20T02:39:48.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.034+0000 7f530b7fe700 1 -- 172.21.15.17:0/4023142067 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f531009a950 con 0x7f531c0ff660 2023-07-20T02:39:48.037 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:48.037 INFO:teuthology.orchestra.run.smithi017.stdout:[ 2023-07-20T02:39:48.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2023-07-20T02:39:48.038 INFO:teuthology.orchestra.run.smithi017.stdout: "poolnum": 1, 2023-07-20T02:39:48.038 INFO:teuthology.orchestra.run.smithi017.stdout: "poolname": ".mgr" 2023-07-20T02:39:48.038 INFO:teuthology.orchestra.run.smithi017.stdout: } 2023-07-20T02:39:48.038 INFO:teuthology.orchestra.run.smithi017.stdout:] 2023-07-20T02:39:48.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f53040a11a0 msgr2=0x7f53040a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:48.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f53040a11a0 0x7f53040a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f530c005950 tx=0x7f530c00b410 comp rx=0 tx=0).stop 2023-07-20T02:39:48.040 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 msgr2=0x7f531c069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:48.041 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c069470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5310009fd0 tx=0x7f531000f690 comp rx=0 tx=0).stop 2023-07-20T02:39:48.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 shutdown_connections 2023-07-20T02:39:48.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f53040a11a0 0x7f53040a3660 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f531c1011f0 0x7f531c19d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f531c0ff660 0x7f531c069470 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 --2- 172.21.15.17:0/4023142067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f531c100850 0x7f531c19b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 >> 172.21.15.17:0/4023142067 conn(0x7f531c0fab90 msgr2=0x7f531c0fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:48.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 shutdown_connections 2023-07-20T02:39:48.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.036+0000 7f5320824700 1 -- 172.21.15.17:0/4023142067 wait complete. 2023-07-20T02:39:48.050 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-20T02:39:48.051 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-20T02:39:48.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 -- 172.21.15.17:0/3079542939 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c1011f0 msgr2=0x7fda5c107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:48.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 --2- 172.21.15.17:0/3079542939 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c1011f0 0x7fda5c107aa0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fda5400b0d0 tx=0x7fda5400b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 -- 172.21.15.17:0/3079542939 shutdown_connections 2023-07-20T02:39:48.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 --2- 172.21.15.17:0/3079542939 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c1011f0 0x7fda5c107aa0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 --2- 172.21.15.17:0/3079542939 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda5c0ff660 0x7fda5c0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 --2- 172.21.15.17:0/3079542939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda5c100850 0x7fda5c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 -- 172.21.15.17:0/3079542939 >> 172.21.15.17:0/3079542939 conn(0x7fda5c0fabd0 msgr2=0x7fda5c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 -- 172.21.15.17:0/3079542939 shutdown_connections 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.183+0000 7fda61bea700 1 -- 172.21.15.17:0/3079542939 wait complete. 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 Processor -- start 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 -- start start 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c0ff660 0x7fda5c196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda5c100850 0x7fda5c196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda5c1011f0 0x7fda5c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c19e0c0 con 0x7fda5c0ff660 2023-07-20T02:39:48.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c19e200 con 0x7fda5c1011f0 2023-07-20T02:39:48.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda61bea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c19e390 con 0x7fda5c100850 2023-07-20T02:39:48.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda5b7fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c0ff660 0x7fda5c196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:48.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda5b7fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c0ff660 0x7fda5c196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43552/0 (socket says 172.21.15.17:43552) 2023-07-20T02:39:48.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda5b7fe700 1 -- 172.21.15.17:0/2368465353 learned_addr learned my addr 172.21.15.17:0/2368465353 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda5bfff700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda5c1011f0 0x7fda5c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.184+0000 7fda5affd700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda5c100850 0x7fda5c196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda5b7fe700 1 -- 172.21.15.17:0/2368465353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda5c100850 msgr2=0x7fda5c196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda5b7fe700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda5c100850 0x7fda5c196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda5b7fe700 1 -- 172.21.15.17:0/2368465353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda5c1011f0 msgr2=0x7fda5c19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda5b7fe700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda5c1011f0 0x7fda5c19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda5b7fe700 1 -- 172.21.15.17:0/2368465353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54009d00 con 0x7fda5c0ff660 2023-07-20T02:39:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda5b7fe700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c0ff660 0x7fda5c196540 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00c970 tx=0x7fda4c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:48.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda58ff9700 1 -- 172.21.15.17:0/2368465353 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c004220 con 0x7fda5c0ff660 2023-07-20T02:39:48.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda58ff9700 1 -- 172.21.15.17:0/2368465353 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda4c007d60 con 0x7fda5c0ff660 2023-07-20T02:39:48.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c19e670 con 0x7fda5c0ff660 2023-07-20T02:39:48.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda58ff9700 1 -- 172.21.15.17:0/2368465353 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c003c90 con 0x7fda5c0ff660 2023-07-20T02:39:48.192 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.185+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c19ebd0 con 0x7fda5c0ff660 2023-07-20T02:39:48.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.186+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c10ae80 con 0x7fda5c0ff660 2023-07-20T02:39:48.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.187+0000 7fda58ff9700 1 -- 172.21.15.17:0/2368465353 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fda4c004380 con 0x7fda5c0ff660 2023-07-20T02:39:48.193 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.188+0000 7fda58ff9700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fda480a11d0 0x7fda480a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:48.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.188+0000 7fda5affd700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fda480a11d0 0x7fda480a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:48.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.188+0000 7fda58ff9700 1 -- 172.21.15.17:0/2368465353 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda4c0cc350 con 0x7fda5c0ff660 2023-07-20T02:39:48.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.189+0000 7fda5affd700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fda480a11d0 0x7fda480a3690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda44009b30 tx=0x7fda44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:48.194 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.192+0000 7fda58ff9700 1 -- 172.21.15.17:0/2368465353 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fda4c095fc0 con 0x7fda5c0ff660 2023-07-20T02:39:48.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:48.401+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fda5c10b040 con 0x7fda5c0ff660 2023-07-20T02:39:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:48 smithi017 ceph-mon[127345]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:48 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2597533508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:48 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3042454752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:48 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/4023142067' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T02:39:48.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:48 smithi146 ceph-mon[128841]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:48.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:48 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2597533508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:48.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:48 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3042454752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:48.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:48 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/4023142067' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T02:39:48.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:48 smithi151 ceph-mon[129163]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:48.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:48 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2597533508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:48.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:48 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3042454752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:39:48.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:48 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/4023142067' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T02:39:49.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.201+0000 7fda58ff9700 1 -- 172.21.15.17:0/2368465353 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7fda4c0995e0 con 0x7fda5c0ff660 2023-07-20T02:39:49.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.204+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fda480a11d0 msgr2=0x7fda480a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:49.205 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.204+0000 7fda61bea700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fda480a11d0 0x7fda480a3690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda44009b30 tx=0x7fda44009260 comp rx=0 tx=0).stop 2023-07-20T02:39:49.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.204+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c0ff660 msgr2=0x7fda5c196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:49.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.204+0000 7fda61bea700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c0ff660 0x7fda5c196540 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00c970 tx=0x7fda4c00cc80 comp rx=0 tx=0).stop 2023-07-20T02:39:49.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 shutdown_connections 2023-07-20T02:39:49.206 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fda480a11d0 0x7fda480a3690 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda5c1011f0 0x7fda5c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fda5c0ff660 0x7fda5c196540 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 --2- 172.21.15.17:0/2368465353 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda5c100850 0x7fda5c196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 >> 172.21.15.17:0/2368465353 conn(0x7fda5c0fabd0 msgr2=0x7fda5c067950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:49.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 shutdown_connections 2023-07-20T02:39:49.207 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.205+0000 7fda61bea700 1 -- 172.21.15.17:0/2368465353 wait complete. 2023-07-20T02:39:49.208 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'cephfs_metadata' created 2023-07-20T02:39:49.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-20T02:39:49.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.374+0000 7f64ace11700 1 -- 172.21.15.17:0/329432325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a8101280 msgr2=0x7f64a8103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:49.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.374+0000 7f64ace11700 1 --2- 172.21.15.17:0/329432325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a8101280 0x7f64a8103670 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64980098f0 tx=0x7f6498009c00 comp rx=0 tx=0).stop 2023-07-20T02:39:49.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.374+0000 7f64ace11700 1 -- 172.21.15.17:0/329432325 shutdown_connections 2023-07-20T02:39:49.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.374+0000 7f64ace11700 1 --2- 172.21.15.17:0/329432325 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a81064e0 0x7f64a81088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.374+0000 7f64ace11700 1 --2- 172.21.15.17:0/329432325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a8101280 0x7f64a8103670 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.374+0000 7f64ace11700 1 --2- 172.21.15.17:0/329432325 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a8103bb0 0x7f64a8105fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.374+0000 7f64ace11700 1 -- 172.21.15.17:0/329432325 >> 172.21.15.17:0/329432325 conn(0x7f64a80fac30 msgr2=0x7f64a80fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:49.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 -- 172.21.15.17:0/329432325 shutdown_connections 2023-07-20T02:39:49.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 -- 172.21.15.17:0/329432325 wait complete. 2023-07-20T02:39:49.379 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 Processor -- start 2023-07-20T02:39:49.379 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 -- start start 2023-07-20T02:39:49.379 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a8101280 0x7f64a8196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:49.379 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a8103bb0 0x7f64a8196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:49.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a81064e0 0x7f64a819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:49.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a819dfd0 con 0x7f64a8103bb0 2023-07-20T02:39:49.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a819e110 con 0x7f64a81064e0 2023-07-20T02:39:49.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.375+0000 7f64ace11700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a819e2a0 con 0x7f64a8101280 2023-07-20T02:39:49.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a8103bb0 0x7f64a8196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:49.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a8103bb0 0x7f64a8196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43568/0 (socket says 172.21.15.17:43568) 2023-07-20T02:39:49.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 -- 172.21.15.17:0/1586999154 learned_addr learned my addr 172.21.15.17:0/1586999154 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:49.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a659c700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a8101280 0x7f64a8196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:49.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a6d9d700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a81064e0 0x7f64a819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:49.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 -- 172.21.15.17:0/1586999154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a8101280 msgr2=0x7f64a8196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:49.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a8101280 0x7f64a8196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 -- 172.21.15.17:0/1586999154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a81064e0 msgr2=0x7f64a819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:49.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a81064e0 0x7f64a819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:49.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 -- 172.21.15.17:0/1586999154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498009520 con 0x7f64a8103bb0 2023-07-20T02:39:49.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a6d9d700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a81064e0 0x7f64a819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:39:49.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64a5d9b700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a8103bb0 0x7f64a8196990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9a0 tx=0x7f649c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:49.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64977fe700 1 -- 172.21.15.17:0/1586999154 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c009790 con 0x7f64a8103bb0 2023-07-20T02:39:49.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a819e580 con 0x7f64a8103bb0 2023-07-20T02:39:49.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64977fe700 1 -- 172.21.15.17:0/1586999154 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f649c010430 con 0x7f64a8103bb0 2023-07-20T02:39:49.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.376+0000 7f64977fe700 1 -- 172.21.15.17:0/1586999154 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c018680 con 0x7f64a8103bb0 2023-07-20T02:39:49.383 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.377+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a819eae0 con 0x7f64a8103bb0 2023-07-20T02:39:49.383 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.377+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a810bf80 con 0x7f64a8103bb0 2023-07-20T02:39:49.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.380+0000 7f64977fe700 1 -- 172.21.15.17:0/1586999154 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f649c0105a0 con 0x7f64a8103bb0 2023-07-20T02:39:49.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.380+0000 7f64977fe700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64900a1150 0x7f64900a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:49.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.380+0000 7f64977fe700 1 -- 172.21.15.17:0/1586999154 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f649c014070 con 0x7f64a8103bb0 2023-07-20T02:39:49.384 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.380+0000 7f64a659c700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64900a1150 0x7f64900a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:49.385 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.381+0000 7f64a659c700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64900a1150 0x7f64900a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f649800fa40 tx=0x7f6498019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:49.385 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.383+0000 7f64977fe700 1 -- 172.21.15.17:0/1586999154 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f649c096410 con 0x7f64a8103bb0 2023-07-20T02:39:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:49 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2368465353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:39:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:49.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:49 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:49.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:49 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2368465353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:39:49.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:49.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:49 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:49.588 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:49.587+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f64a819f450 con 0x7f64a8103bb0 2023-07-20T02:39:49.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:49 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2368465353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:39:49.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:49.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:49 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.210 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.207+0000 7f64977fe700 1 -- 172.21.15.17:0/1586999154 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7f649c099a30 con 0x7f64a8103bb0 2023-07-20T02:39:50.223 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64900a1150 msgr2=0x7f64900a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:50.223 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64900a1150 0x7f64900a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f649800fa40 tx=0x7f6498019040 comp rx=0 tx=0).stop 2023-07-20T02:39:50.223 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a8103bb0 msgr2=0x7f64a8196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:50.223 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a8103bb0 0x7f64a8196990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9a0 tx=0x7f649c00ecb0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.223 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 shutdown_connections 2023-07-20T02:39:50.223 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64900a1150 0x7f64900a3610 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f649800fa40 tx=0x7f6498019040 comp rx=0 tx=0).stop 2023-07-20T02:39:50.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a81064e0 0x7f64a819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a8101280 0x7f64a8196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 --2- 172.21.15.17:0/1586999154 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a8103bb0 0x7f64a8196990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.217+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 >> 172.21.15.17:0/1586999154 conn(0x7f64a80fac30 msgr2=0x7f64a8102760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:50.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.220+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 shutdown_connections 2023-07-20T02:39:50.224 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.220+0000 7f64ace11700 1 -- 172.21.15.17:0/1586999154 wait complete. 2023-07-20T02:39:50.225 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'cephfs_data' created 2023-07-20T02:39:50.235 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-20T02:39:50.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.368+0000 7f16ef64a700 1 -- 172.21.15.17:0/4079413959 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e8103b10 msgr2=0x7f16e8105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:50.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.368+0000 7f16ef64a700 1 --2- 172.21.15.17:0/4079413959 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e8103b10 0x7f16e8105f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009990 tx=0x7f16d8009ca0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.372 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.368+0000 7f16ef64a700 1 -- 172.21.15.17:0/4079413959 shutdown_connections 2023-07-20T02:39:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.368+0000 7f16ef64a700 1 --2- 172.21.15.17:0/4079413959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16e8106440 0x7f16e81088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.368+0000 7f16ef64a700 1 --2- 172.21.15.17:0/4079413959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e81011e0 0x7f16e81035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.368+0000 7f16ef64a700 1 --2- 172.21.15.17:0/4079413959 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e8103b10 0x7f16e8105f00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.368+0000 7f16ef64a700 1 -- 172.21.15.17:0/4079413959 >> 172.21.15.17:0/4079413959 conn(0x7f16e80fabd0 msgr2=0x7f16e80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.369+0000 7f16ef64a700 1 -- 172.21.15.17:0/4079413959 shutdown_connections 2023-07-20T02:39:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.369+0000 7f16ef64a700 1 -- 172.21.15.17:0/4079413959 wait complete. 2023-07-20T02:39:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.369+0000 7f16ef64a700 1 Processor -- start 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.369+0000 7f16ef64a700 1 -- start start 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.369+0000 7f16ef64a700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e81011e0 0x7f16e81964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ef64a700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16e8103b10 0x7f16e8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ef64a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e8106440 0x7f16e819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ef64a700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e819e040 con 0x7f16e81011e0 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ef64a700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e819e180 con 0x7f16e8103b10 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ef64a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e819e310 con 0x7f16e8106440 2023-07-20T02:39:50.374 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e81011e0 0x7f16e81964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:50.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e81011e0 0x7f16e81964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43584/0 (socket says 172.21.15.17:43584) 2023-07-20T02:39:50.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 -- 172.21.15.17:0/1219499727 learned_addr learned my addr 172.21.15.17:0/1219499727 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:50.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16edbe7700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e8106440 0x7f16e819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:50.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ecbe5700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16e8103b10 0x7f16e8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:50.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 -- 172.21.15.17:0/1219499727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e8106440 msgr2=0x7f16e819da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:50.375 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e8106440 0x7f16e819da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 -- 172.21.15.17:0/1219499727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16e8103b10 msgr2=0x7f16e8196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:50.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16e8103b10 0x7f16e8196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:50.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 -- 172.21.15.17:0/1219499727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d80095f0 con 0x7f16e81011e0 2023-07-20T02:39:50.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ed3e6700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e81011e0 0x7f16e81964c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f16e400e9a0 tx=0x7f16e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:50.376 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16de7fc700 1 -- 172.21.15.17:0/1219499727 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4009790 con 0x7f16e81011e0 2023-07-20T02:39:50.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16de7fc700 1 -- 172.21.15.17:0/1219499727 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16e4010430 con 0x7f16e81011e0 2023-07-20T02:39:50.377 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e819e5f0 con 0x7f16e81011e0 2023-07-20T02:39:50.378 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16de7fc700 1 -- 172.21.15.17:0/1219499727 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4018680 con 0x7f16e81011e0 2023-07-20T02:39:50.378 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.370+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e819eb00 con 0x7f16e81011e0 2023-07-20T02:39:50.379 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.371+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e810bfa0 con 0x7f16e81011e0 2023-07-20T02:39:50.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.373+0000 7f16de7fc700 1 -- 172.21.15.17:0/1219499727 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f16e40105a0 con 0x7f16e81011e0 2023-07-20T02:39:50.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.373+0000 7f16de7fc700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f16d40a1150 0x7f16d40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:50.380 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.373+0000 7f16ecbe5700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f16d40a1150 0x7f16d40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:50.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.374+0000 7f16de7fc700 1 -- 172.21.15.17:0/1219499727 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f16e4014070 con 0x7f16e81011e0 2023-07-20T02:39:50.381 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.374+0000 7f16ecbe5700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f16d40a1150 0x7f16d40a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f16d800b2e0 tx=0x7f16d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:50.382 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.381+0000 7f16de7fc700 1 -- 172.21.15.17:0/1219499727 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f16e4096410 con 0x7f16e81011e0 2023-07-20T02:39:50.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:50.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2368465353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T02:39:50.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T02:39:50.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1586999154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T02:39:50.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:50.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:50.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:50 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:50.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2368465353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T02:39:50.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T02:39:50.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1586999154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:50.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:50 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.657 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:50.656+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f16e810c130 con 0x7f16e81011e0 2023-07-20T02:39:50.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:50.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2368465353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T02:39:50.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1586999154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:50.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:50 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:51.239 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.232+0000 7f16de7fc700 1 -- 172.21.15.17:0/1219499727 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f16e40260c0 con 0x7f16e81011e0 2023-07-20T02:39:51.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f16d40a1150 msgr2=0x7f16d40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f16d40a1150 0x7f16d40a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f16d800b2e0 tx=0x7f16d8005fb0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e81011e0 msgr2=0x7f16e81964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e81011e0 0x7f16e81964c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f16e400e9a0 tx=0x7f16e400ecb0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 shutdown_connections 2023-07-20T02:39:51.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f16d40a1150 0x7f16d40a3610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16e8106440 0x7f16e819da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f16e81011e0 0x7f16e81964c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.244 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 --2- 172.21.15.17:0/1219499727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16e8103b10 0x7f16e8196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.244 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.241+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 >> 172.21.15.17:0/1219499727 conn(0x7f16e80fabd0 msgr2=0x7f16e8102690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:51.245 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.244+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 shutdown_connections 2023-07-20T02:39:51.245 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.244+0000 7f16ef64a700 1 -- 172.21.15.17:0/1219499727 wait complete. 2023-07-20T02:39:51.246 INFO:teuthology.orchestra.run.smithi017.stderr:new fs with metadata pool 2 and data pool 3 2023-07-20T02:39:51.258 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:39:51.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 -- 172.21.15.17:0/3119253873 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 msgr2=0x7f1964107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 --2- 172.21.15.17:0/3119253873 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f1964107a90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f195c00b0d0 tx=0x7f195c00b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 -- 172.21.15.17:0/3119253873 shutdown_connections 2023-07-20T02:39:51.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 --2- 172.21.15.17:0/3119253873 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f1964107a90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 --2- 172.21.15.17:0/3119253873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 0x7f19640ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 --2- 172.21.15.17:0/3119253873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1964100840 0x7f1964100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 -- 172.21.15.17:0/3119253873 >> 172.21.15.17:0/3119253873 conn(0x7f19640faba0 msgr2=0x7f19640fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:51.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 -- 172.21.15.17:0/3119253873 shutdown_connections 2023-07-20T02:39:51.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.390+0000 7f19697e7700 1 -- 172.21.15.17:0/3119253873 wait complete. 2023-07-20T02:39:51.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.391+0000 7f19697e7700 1 Processor -- start 2023-07-20T02:39:51.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.391+0000 7f19697e7700 1 -- start start 2023-07-20T02:39:51.396 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.391+0000 7f19697e7700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 0x7f1964068440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.391+0000 7f19697e7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1964100840 0x7f196419d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.391+0000 7f19697e7700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f196419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19697e7700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19641a0280 con 0x7f19641011e0 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19697e7700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19641a03f0 con 0x7f19640ff650 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19697e7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19641a05b0 con 0x7f1964100840 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f196419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f1962ffd700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 0x7f1964068440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.397 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f196419fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43604/0 (socket says 172.21.15.17:43604) 2023-07-20T02:39:51.398 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 -- 172.21.15.17:0/3016380686 learned_addr learned my addr 172.21.15.17:0/3016380686 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:51.398 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f1962ffd700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 0x7f1964068440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.17:48576/0 (socket says 172.21.15.17:48576) 2023-07-20T02:39:51.398 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19627fc700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1964100840 0x7f196419d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.398 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 -- 172.21.15.17:0/3016380686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1964100840 msgr2=0x7f196419d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.398 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1964100840 0x7f196419d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.398 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 -- 172.21.15.17:0/3016380686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 msgr2=0x7f1964068440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.398 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 0x7f1964068440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 -- 172.21.15.17:0/3016380686 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c009d00 con 0x7f19641011e0 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f1962ffd700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 0x7f1964068440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19637fe700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f196419fbd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f195c007990 tx=0x7f195c008eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f195bfff700 1 -- 172.21.15.17:0/3016380686 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c00e040 con 0x7f19641011e0 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19641a0830 con 0x7f19641011e0 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f195bfff700 1 -- 172.21.15.17:0/3016380686 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f195c00bdb0 con 0x7f19641011e0 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.392+0000 7f195bfff700 1 -- 172.21.15.17:0/3016380686 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c01d470 con 0x7f19641011e0 2023-07-20T02:39:51.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.393+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19641a0d00 con 0x7f19641011e0 2023-07-20T02:39:51.400 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.393+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1964106b70 con 0x7f19641011e0 2023-07-20T02:39:51.400 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.395+0000 7f195bfff700 1 -- 172.21.15.17:0/3016380686 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f195c0050b0 con 0x7f19641011e0 2023-07-20T02:39:51.400 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.396+0000 7f195bfff700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f19500a11a0 0x7f19500a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.401 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.396+0000 7f195bfff700 1 -- 172.21.15.17:0/3016380686 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f195c0cddf0 con 0x7f19641011e0 2023-07-20T02:39:51.401 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.396+0000 7f1962ffd700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f19500a11a0 0x7f19500a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.401 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.397+0000 7f1962ffd700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f19500a11a0 0x7f19500a3660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19641006a0 tx=0x7f1954009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:51.401 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.400+0000 7f195bfff700 1 -- 172.21.15.17:0/3016380686 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f195c097720 con 0x7f19641011e0 2023-07-20T02:39:51.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:51 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1586999154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T02:39:51.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:51 smithi146 ceph-mon[128841]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T02:39:51.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:51 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:51.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:51 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1219499727' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:39:51.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:51 smithi146 ceph-mon[128841]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T02:39:51.606 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.605+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f196402cb40 con 0x7f19641011e0 2023-07-20T02:39:51.607 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.606+0000 7f195bfff700 1 -- 172.21.15.17:0/3016380686 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22131 (secure 0 0 0) 0x7f195c09ad40 con 0x7f19641011e0 2023-07-20T02:39:51.608 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:51.608 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":82,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","created":"2023-07-20T02:32:50.175394+0000","modified":"2023-07-20T02:39:50.656618+0000","last_up_change":"2023-07-20T02:39:09.254013+0000","last_in_change":"2023-07-20T02:38:51.253472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:35:55.360132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:39:48.842354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:39:50.025623+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb89ee2e-e4ff-4c83-9626-69cbfb526dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6803","nonce":795351069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6805","nonce":795351069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6809","nonce":795351069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6807","nonce":795351069}]},"public_addr":"172.21.15.17:6803/795351069","cluster_addr":"172.21.15.17:6805/795351069","heartbeat_back_addr":"172.21.15.17:6809/795351069","heartbeat_front_addr":"172.21.15.17:6807/795351069","state":["exists","up"]},{"osd":1,"uuid":"2d422389-0925-410c-bd28-50ad06d68cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6801","nonce":3123864696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6803","nonce":3123864696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6807","nonce":3123864696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6805","nonce":3123864696}]},"public_addr":"172.21.15.146:6801/3123864696","cluster_addr":"172.21.15.146:6803/3123864696","heartbeat_back_addr":"172.21.15.146:6807/3123864696","heartbeat_front_addr":"172.21.15.146:6805/3123864696","state":["exists","up"]},{"osd":2,"uuid":"a08d37e7-a3cd-4694-9441-b505bd167c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6801","nonce":1299864358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6803","nonce":1299864358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6807","nonce":1299864358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6805","nonce":1299864358}]},"public_addr":"172.21.15.151:6801/1299864358","cluster_addr":"172.21.15.151:6803/1299864358","heartbeat_back_addr":"172.21.15.151:6807/1299864358","heartbeat_front_addr":"172.21.15.151:6805/1299864358","state":["exists","up"]},{"osd":3,"uuid":"62ad61a1-05b0-4e38-a2eb-530485d806b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6811","nonce":604342850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6813","nonce":604342850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6817","nonce":604342850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6815","nonce":604342850}]},"public_addr":"172.21.15.17:6811/604342850","cluster_addr":"172.21.15.17:6813/604342850","heartbeat_back_addr":"172.21.15.17:6817/604342850","heartbeat_front_addr":"172.21.15.17:6815/604342850","state":["exists","up"]},{"osd":4,"uuid":"38ed288f-9c0d-415c-b52f-075d3b7599db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6809","nonce":4024595090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6811","nonce":4024595090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6815","nonce":4024595090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6813","nonce":4024595090}]},"public_addr":"172.21.15.146:6809/4024595090","cluster_addr":"172.21.15.146:6811/4024595090","heartbeat_back_addr":"172.21.15.146:6815/4024595090","heartbeat_front_addr":"172.21.15.146:6813/4024595090","state":["exists","up"]},{"osd":5,"uuid":"7da194cb-67fb-4c73-b84c-fa899c3635b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6809","nonce":1633219838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6811","nonce":1633219838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6815","nonce":1633219838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6813","nonce":1633219838}]},"public_addr":"172.21.15.151:6809/1633219838","cluster_addr":"172.21.15.151:6811/1633219838","heartbeat_back_addr":"172.21.15.151:6815/1633219838","heartbeat_front_addr":"172.21.15.151:6813/1633219838","state":["exists","up"]},{"osd":6,"uuid":"67ced483-d37a-46c1-97ae-288f28b091f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6819","nonce":3807389822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6821","nonce":3807389822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6825","nonce":3807389822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6823","nonce":3807389822}]},"public_addr":"172.21.15.17:6819/3807389822","cluster_addr":"172.21.15.17:6821/3807389822","heartbeat_back_addr":"172.21.15.17:6825/3807389822","heartbeat_front_addr":"172.21.15.17:6823/3807389822","state":["exists","up"]},{"osd":7,"uuid":"26199bbc-d82f-4e97-9218-0058426ef412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6817","nonce":708669673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6819","nonce":708669673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6823","nonce":708669673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6821","nonce":708669673}]},"public_addr":"172.21.15.146:6817/708669673","cluster_addr":"172.21.15.146:6819/708669673","heartbeat_back_addr":"172.21.15.146:6823/708669673","heartbeat_front_addr":"172.21.15.146:6821/708669673","state":["exists","up"]},{"osd":8,"uuid":"b1a33778-9ab5-45db-90ff-644794ee1cd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6817","nonce":2042667088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6819","nonce":2042667088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6823","nonce":2042667088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6821","nonce":2042667088}]},"public_addr":"172.21.15.151:6817/2042667088","cluster_addr":"172.21.15.151:6819/2042667088","heartbeat_back_addr":"172.21.15.151:6823/2042667088","heartbeat_front_addr":"172.21.15.151:6821/2042667088","state":["exists","up"]},{"osd":9,"uuid":"496d04b3-61a4-4405-aa52-5590aacf7129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6827","nonce":1403136476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6829","nonce":1403136476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6833","nonce":1403136476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6831","nonce":1403136476}]},"public_addr":"172.21.15.17:6827/1403136476","cluster_addr":"172.21.15.17:6829/1403136476","heartbeat_back_addr":"172.21.15.17:6833/1403136476","heartbeat_front_addr":"172.21.15.17:6831/1403136476","state":["exists","up"]},{"osd":10,"uuid":"6ee41a91-3ac3-4262-bff2-7531048859c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6825","nonce":2995236638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6827","nonce":2995236638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6831","nonce":2995236638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6829","nonce":2995236638}]},"public_addr":"172.21.15.146:6825/2995236638","cluster_addr":"172.21.15.146:6827/2995236638","heartbeat_back_addr":"172.21.15.146:6831/2995236638","heartbeat_front_addr":"172.21.15.146:6829/2995236638","state":["exists","up"]},{"osd":11,"uuid":"282d07a4-1bb0-4239-9c1e-adeebe9d7432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6825","nonce":1663985079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6827","nonce":1663985079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6831","nonce":1663985079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6829","nonce":1663985079}]},"public_addr":"172.21.15.151:6825/1663985079","cluster_addr":"172.21.15.151:6827/1663985079","heartbeat_back_addr":"172.21.15.151:6831/1663985079","heartbeat_front_addr":"172.21.15.151:6829/1663985079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:07.560935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:29.818964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:51.802583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:12.985262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:34.792736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:56.378329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:17.498840+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:40.421290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:02.192805+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:23.342606+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:45.695709+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:39:07.701215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2979320785":"2023-07-21T02:33:44.823257+0000","172.21.15.17:0/979357839":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6800/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/714275157":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1888486288":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/910909892":"2023-07-21T02:33:19.367142+0000","172.21.15.17:6800/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1536507180":"2023-07-21T02:33:44.823257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:39:51.609 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f19500a11a0 msgr2=0x7f19500a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f19500a11a0 0x7f19500a3660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19641006a0 tx=0x7f1954009260 comp rx=0 tx=0).stop 2023-07-20T02:39:51.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 msgr2=0x7f196419fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f196419fbd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f195c007990 tx=0x7f195c008eb0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 shutdown_connections 2023-07-20T02:39:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f19500a11a0 0x7f19500a3660 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f19641011e0 0x7f196419fbd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19640ff650 0x7f1964068440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 --2- 172.21.15.17:0/3016380686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1964100840 0x7f196419d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.608+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 >> 172.21.15.17:0/3016380686 conn(0x7f19640faba0 msgr2=0x7f1964103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.609+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 shutdown_connections 2023-07-20T02:39:51.611 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.609+0000 7f19697e7700 1 -- 172.21.15.17:0/3016380686 wait complete. 2023-07-20T02:39:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:51 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1586999154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T02:39:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:51 smithi151 ceph-mon[129163]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T02:39:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:51 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:51 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1219499727' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:39:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:51 smithi151 ceph-mon[129163]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T02:39:51.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:51 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1586999154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T02:39:51.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:51 smithi017 ceph-mon[127345]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T02:39:51.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:51 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:39:51.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:51 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1219499727' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:39:51.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:51 smithi017 ceph-mon[127345]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T02:39:51.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:39:51.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 -- 172.21.15.17:0/1616148519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd4100850 msgr2=0x7f8fd4100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 --2- 172.21.15.17:0/1616148519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd4100850 0x7f8fd4100cb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80099c0 tx=0x7f8fc8009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 -- 172.21.15.17:0/1616148519 shutdown_connections 2023-07-20T02:39:51.755 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 --2- 172.21.15.17:0/1616148519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd41011f0 0x7f8fd4107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 --2- 172.21.15.17:0/1616148519 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fd40ff660 0x7f8fd40ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 --2- 172.21.15.17:0/1616148519 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd4100850 0x7f8fd4100cb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 -- 172.21.15.17:0/1616148519 >> 172.21.15.17:0/1616148519 conn(0x7f8fd40fabd0 msgr2=0x7f8fd40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:51.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 -- 172.21.15.17:0/1616148519 shutdown_connections 2023-07-20T02:39:51.756 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.753+0000 7f8fd984f700 1 -- 172.21.15.17:0/1616148519 wait complete. 2023-07-20T02:39:51.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.754+0000 7f8fd984f700 1 Processor -- start 2023-07-20T02:39:51.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.754+0000 7f8fd984f700 1 -- start start 2023-07-20T02:39:51.758 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd984f700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd40ff660 0x7f8fd4069680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd984f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd4100850 0x7f8fd419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd984f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fd41011f0 0x7f8fd419d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd984f700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd419e000 con 0x7f8fd40ff660 2023-07-20T02:39:51.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd984f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd419e170 con 0x7f8fd4100850 2023-07-20T02:39:51.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd984f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd419e330 con 0x7f8fd41011f0 2023-07-20T02:39:51.759 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd40ff660 0x7f8fd4069680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd40ff660 0x7f8fd4069680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43614/0 (socket says 172.21.15.17:43614) 2023-07-20T02:39:51.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 -- 172.21.15.17:0/3205447543 learned_addr learned my addr 172.21.15.17:0/3205447543 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:51.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd27fc700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd4100850 0x7f8fd419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd37fe700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fd41011f0 0x7f8fd419d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 -- 172.21.15.17:0/3205447543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fd41011f0 msgr2=0x7f8fd419d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.760 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fd41011f0 0x7f8fd419d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 -- 172.21.15.17:0/3205447543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd4100850 msgr2=0x7f8fd419b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd4100850 0x7f8fd419b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 -- 172.21.15.17:0/3205447543 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc80095f0 con 0x7f8fd40ff660 2023-07-20T02:39:51.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd2ffd700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd40ff660 0x7f8fd4069680 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400b9c0 tx=0x7f8fc400bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:51.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.755+0000 7f8fd884d700 1 -- 172.21.15.17:0/3205447543 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400d8e0 con 0x7f8fd40ff660 2023-07-20T02:39:51.761 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.756+0000 7f8fd884d700 1 -- 172.21.15.17:0/3205447543 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fc4009900 con 0x7f8fd40ff660 2023-07-20T02:39:51.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.756+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd419e610 con 0x7f8fd40ff660 2023-07-20T02:39:51.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.756+0000 7f8fd884d700 1 -- 172.21.15.17:0/3205447543 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400c400 con 0x7f8fd40ff660 2023-07-20T02:39:51.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.756+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd419eaf0 con 0x7f8fd40ff660 2023-07-20T02:39:51.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.757+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd410af60 con 0x7f8fd40ff660 2023-07-20T02:39:51.762 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.758+0000 7f8fd884d700 1 -- 172.21.15.17:0/3205447543 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8fc400da40 con 0x7f8fd40ff660 2023-07-20T02:39:51.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.758+0000 7f8fd884d700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f8fbc0a1220 0x7f8fbc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:51.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.758+0000 7f8fd27fc700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f8fbc0a1220 0x7f8fbc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:51.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.759+0000 7f8fd884d700 1 -- 172.21.15.17:0/3205447543 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8fc4011070 con 0x7f8fd40ff660 2023-07-20T02:39:51.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.759+0000 7f8fd27fc700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f8fbc0a1220 0x7f8fbc0a36e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800b580 tx=0x7f8fc8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:51.763 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.762+0000 7f8fd884d700 1 -- 172.21.15.17:0/3205447543 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8fc404e780 con 0x7f8fd40ff660 2023-07-20T02:39:51.974 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.973+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd419f130 con 0x7f8fd40ff660 2023-07-20T02:39:51.976 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.975+0000 7f8fd884d700 1 -- 172.21.15.17:0/3205447543 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22131 (secure 0 0 0) 0x7f8fc4099660 con 0x7f8fd40ff660 2023-07-20T02:39:51.976 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-07-20T02:39:51.976 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":82,"fsid":"9d6085e8-26a5-11ee-9b34-001a4aab830c","created":"2023-07-20T02:32:50.175394+0000","modified":"2023-07-20T02:39:50.656618+0000","last_up_change":"2023-07-20T02:39:09.254013+0000","last_in_change":"2023-07-20T02:38:51.253472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:35:55.360132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:39:48.842354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:39:50.025623+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb89ee2e-e4ff-4c83-9626-69cbfb526dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6803","nonce":795351069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6805","nonce":795351069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6809","nonce":795351069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":795351069},{"type":"v1","addr":"172.21.15.17:6807","nonce":795351069}]},"public_addr":"172.21.15.17:6803/795351069","cluster_addr":"172.21.15.17:6805/795351069","heartbeat_back_addr":"172.21.15.17:6809/795351069","heartbeat_front_addr":"172.21.15.17:6807/795351069","state":["exists","up"]},{"osd":1,"uuid":"2d422389-0925-410c-bd28-50ad06d68cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6801","nonce":3123864696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6803","nonce":3123864696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6807","nonce":3123864696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3123864696},{"type":"v1","addr":"172.21.15.146:6805","nonce":3123864696}]},"public_addr":"172.21.15.146:6801/3123864696","cluster_addr":"172.21.15.146:6803/3123864696","heartbeat_back_addr":"172.21.15.146:6807/3123864696","heartbeat_front_addr":"172.21.15.146:6805/3123864696","state":["exists","up"]},{"osd":2,"uuid":"a08d37e7-a3cd-4694-9441-b505bd167c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6801","nonce":1299864358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6803","nonce":1299864358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6807","nonce":1299864358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1299864358},{"type":"v1","addr":"172.21.15.151:6805","nonce":1299864358}]},"public_addr":"172.21.15.151:6801/1299864358","cluster_addr":"172.21.15.151:6803/1299864358","heartbeat_back_addr":"172.21.15.151:6807/1299864358","heartbeat_front_addr":"172.21.15.151:6805/1299864358","state":["exists","up"]},{"osd":3,"uuid":"62ad61a1-05b0-4e38-a2eb-530485d806b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6811","nonce":604342850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6813","nonce":604342850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6817","nonce":604342850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":604342850},{"type":"v1","addr":"172.21.15.17:6815","nonce":604342850}]},"public_addr":"172.21.15.17:6811/604342850","cluster_addr":"172.21.15.17:6813/604342850","heartbeat_back_addr":"172.21.15.17:6817/604342850","heartbeat_front_addr":"172.21.15.17:6815/604342850","state":["exists","up"]},{"osd":4,"uuid":"38ed288f-9c0d-415c-b52f-075d3b7599db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6809","nonce":4024595090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6811","nonce":4024595090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6815","nonce":4024595090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4024595090},{"type":"v1","addr":"172.21.15.146:6813","nonce":4024595090}]},"public_addr":"172.21.15.146:6809/4024595090","cluster_addr":"172.21.15.146:6811/4024595090","heartbeat_back_addr":"172.21.15.146:6815/4024595090","heartbeat_front_addr":"172.21.15.146:6813/4024595090","state":["exists","up"]},{"osd":5,"uuid":"7da194cb-67fb-4c73-b84c-fa899c3635b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6809","nonce":1633219838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6811","nonce":1633219838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6815","nonce":1633219838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1633219838},{"type":"v1","addr":"172.21.15.151:6813","nonce":1633219838}]},"public_addr":"172.21.15.151:6809/1633219838","cluster_addr":"172.21.15.151:6811/1633219838","heartbeat_back_addr":"172.21.15.151:6815/1633219838","heartbeat_front_addr":"172.21.15.151:6813/1633219838","state":["exists","up"]},{"osd":6,"uuid":"67ced483-d37a-46c1-97ae-288f28b091f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6819","nonce":3807389822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6821","nonce":3807389822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6825","nonce":3807389822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3807389822},{"type":"v1","addr":"172.21.15.17:6823","nonce":3807389822}]},"public_addr":"172.21.15.17:6819/3807389822","cluster_addr":"172.21.15.17:6821/3807389822","heartbeat_back_addr":"172.21.15.17:6825/3807389822","heartbeat_front_addr":"172.21.15.17:6823/3807389822","state":["exists","up"]},{"osd":7,"uuid":"26199bbc-d82f-4e97-9218-0058426ef412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6817","nonce":708669673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6819","nonce":708669673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6823","nonce":708669673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":708669673},{"type":"v1","addr":"172.21.15.146:6821","nonce":708669673}]},"public_addr":"172.21.15.146:6817/708669673","cluster_addr":"172.21.15.146:6819/708669673","heartbeat_back_addr":"172.21.15.146:6823/708669673","heartbeat_front_addr":"172.21.15.146:6821/708669673","state":["exists","up"]},{"osd":8,"uuid":"b1a33778-9ab5-45db-90ff-644794ee1cd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6817","nonce":2042667088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6819","nonce":2042667088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6823","nonce":2042667088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2042667088},{"type":"v1","addr":"172.21.15.151:6821","nonce":2042667088}]},"public_addr":"172.21.15.151:6817/2042667088","cluster_addr":"172.21.15.151:6819/2042667088","heartbeat_back_addr":"172.21.15.151:6823/2042667088","heartbeat_front_addr":"172.21.15.151:6821/2042667088","state":["exists","up"]},{"osd":9,"uuid":"496d04b3-61a4-4405-aa52-5590aacf7129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6827","nonce":1403136476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6829","nonce":1403136476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6833","nonce":1403136476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":1403136476},{"type":"v1","addr":"172.21.15.17:6831","nonce":1403136476}]},"public_addr":"172.21.15.17:6827/1403136476","cluster_addr":"172.21.15.17:6829/1403136476","heartbeat_back_addr":"172.21.15.17:6833/1403136476","heartbeat_front_addr":"172.21.15.17:6831/1403136476","state":["exists","up"]},{"osd":10,"uuid":"6ee41a91-3ac3-4262-bff2-7531048859c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6825","nonce":2995236638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6827","nonce":2995236638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6831","nonce":2995236638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2995236638},{"type":"v1","addr":"172.21.15.146:6829","nonce":2995236638}]},"public_addr":"172.21.15.146:6825/2995236638","cluster_addr":"172.21.15.146:6827/2995236638","heartbeat_back_addr":"172.21.15.146:6831/2995236638","heartbeat_front_addr":"172.21.15.146:6829/2995236638","state":["exists","up"]},{"osd":11,"uuid":"282d07a4-1bb0-4239-9c1e-adeebe9d7432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6825","nonce":1663985079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6827","nonce":1663985079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6831","nonce":1663985079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1663985079},{"type":"v1","addr":"172.21.15.151:6829","nonce":1663985079}]},"public_addr":"172.21.15.151:6825/1663985079","cluster_addr":"172.21.15.151:6827/1663985079","heartbeat_back_addr":"172.21.15.151:6831/1663985079","heartbeat_front_addr":"172.21.15.151:6829/1663985079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:07.560935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:29.818964+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:35:51.802583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:12.985262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:34.792736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:36:56.378329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:17.498840+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:37:40.421290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:02.192805+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:23.342606+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:38:45.695709+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:39:07.701215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2979320785":"2023-07-21T02:33:44.823257+0000","172.21.15.17:0/979357839":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6800/3625702005":"2023-07-21T02:33:44.823257+0000","172.21.15.17:6801/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/714275157":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1888486288":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/910909892":"2023-07-21T02:33:19.367142+0000","172.21.15.17:6800/1718815307":"2023-07-21T02:33:19.367142+0000","172.21.15.17:0/1536507180":"2023-07-21T02:33:44.823257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:39:51.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f8fbc0a1220 msgr2=0x7f8fbc0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.978 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f8fbc0a1220 0x7f8fbc0a36e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800b580 tx=0x7f8fc8005fb0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd40ff660 msgr2=0x7f8fd4069680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:51.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd40ff660 0x7f8fd4069680 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400b9c0 tx=0x7f8fc400bcd0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 shutdown_connections 2023-07-20T02:39:51.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f8fbc0a1220 0x7f8fbc0a36e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fd41011f0 0x7f8fd419d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.979 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8fd40ff660 0x7f8fd4069680 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 --2- 172.21.15.17:0/3205447543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd4100850 0x7f8fd419b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:51.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 >> 172.21.15.17:0/3205447543 conn(0x7f8fd40fabd0 msgr2=0x7f8fd40fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:51.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 shutdown_connections 2023-07-20T02:39:51.980 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:51.977+0000 7f8fd984f700 1 -- 172.21.15.17:0/3205447543 wait complete. 2023-07-20T02:39:51.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-20T02:39:52.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 -- 172.21.15.17:0/841255494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 msgr2=0x7fc468107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:52.124 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/841255494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc468107b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc46000b0d0 tx=0x7fc46000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:52.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 -- 172.21.15.17:0/841255494 shutdown_connections 2023-07-20T02:39:52.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/841255494 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc468107b00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:52.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/841255494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4680ff6c0 0x7fc4680ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:52.125 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/841255494 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4681008b0 0x7fc468100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:52.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 -- 172.21.15.17:0/841255494 >> 172.21.15.17:0/841255494 conn(0x7fc4680fac70 msgr2=0x7fc4680fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:52.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 -- 172.21.15.17:0/841255494 shutdown_connections 2023-07-20T02:39:52.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 -- 172.21.15.17:0/841255494 wait complete. 2023-07-20T02:39:52.126 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 Processor -- start 2023-07-20T02:39:52.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.123+0000 7fc46d8ab700 1 -- start start 2023-07-20T02:39:52.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46d8ab700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4680ff6c0 0x7fc468103650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:52.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46d8ab700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4681008b0 0x7fc468101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:52.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46d8ab700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc4681021d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:52.128 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46d8ab700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc468102970 con 0x7fc468101250 2023-07-20T02:39:52.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46d8ab700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc468102ae0 con 0x7fc4680ff6c0 2023-07-20T02:39:52.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46d8ab700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4681a05d0 con 0x7fc4681008b0 2023-07-20T02:39:52.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc4681021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:52.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc4681021d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43632/0 (socket says 172.21.15.17:43632) 2023-07-20T02:39:52.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 -- 172.21.15.17:0/2273283045 learned_addr learned my addr 172.21.15.17:0/2273283045 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:52.129 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc466ffd700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4680ff6c0 0x7fc468103650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:52.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4667fc700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4681008b0 0x7fc468101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:52.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 -- 172.21.15.17:0/2273283045 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4681008b0 msgr2=0x7fc468101c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:52.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4681008b0 0x7fc468101c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:52.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 -- 172.21.15.17:0/2273283045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4680ff6c0 msgr2=0x7fc468103650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:52.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4680ff6c0 0x7fc468103650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:52.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 -- 172.21.15.17:0/2273283045 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460009d00 con 0x7fc468101250 2023-07-20T02:39:52.130 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc4677fe700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc4681021d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc460007990 tx=0x7fc460008eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:52.131 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46c8a9700 1 -- 172.21.15.17:0/2273283045 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000e040 con 0x7fc468101250 2023-07-20T02:39:52.131 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46c8a9700 1 -- 172.21.15.17:0/2273283045 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc46000bdb0 con 0x7fc468101250 2023-07-20T02:39:52.131 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4681a0850 con 0x7fc468101250 2023-07-20T02:39:52.131 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.124+0000 7fc46c8a9700 1 -- 172.21.15.17:0/2273283045 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46001d470 con 0x7fc468101250 2023-07-20T02:39:52.131 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.125+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4681a0cd0 con 0x7fc468101250 2023-07-20T02:39:52.132 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.125+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46810aec0 con 0x7fc468101250 2023-07-20T02:39:52.132 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.127+0000 7fc46c8a9700 1 -- 172.21.15.17:0/2273283045 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc4600050b0 con 0x7fc468101250 2023-07-20T02:39:52.133 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.128+0000 7fc46c8a9700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc4500a1150 0x7fc4500a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:52.133 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.128+0000 7fc46c8a9700 1 -- 172.21.15.17:0/2273283045 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc4600cde90 con 0x7fc468101250 2023-07-20T02:39:52.133 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.128+0000 7fc466ffd700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc4500a1150 0x7fc4500a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:52.133 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.128+0000 7fc466ffd700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc4500a1150 0x7fc4500a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc468100710 tx=0x7fc458009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:52.133 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.131+0000 7fc46c8a9700 1 -- 172.21.15.17:0/2273283045 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc460097720 con 0x7fc468101250 2023-07-20T02:39:52.399 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:52.397+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc46810b050 con 0x7fc468101250 2023-07-20T02:39:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:52 smithi017 ceph-mon[127345]: pgmap v254: 97 pgs: 27 creating+peering, 3 active+clean, 67 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:52 smithi017 ceph-mon[127345]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T02:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:52 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1219499727' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T02:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:52 smithi017 ceph-mon[127345]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T02:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:52 smithi017 ceph-mon[127345]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T02:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:52 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3016380686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:52 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/3205447543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:52.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:52 smithi146 ceph-mon[128841]: pgmap v254: 97 pgs: 27 creating+peering, 3 active+clean, 67 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:52.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:52 smithi146 ceph-mon[128841]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T02:39:52.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:52 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1219499727' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T02:39:52.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:52 smithi146 ceph-mon[128841]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T02:39:52.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:52 smithi146 ceph-mon[128841]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T02:39:52.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:52 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3016380686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:52.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:52 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/3205447543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:52.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:52 smithi151 ceph-mon[129163]: pgmap v254: 97 pgs: 27 creating+peering, 3 active+clean, 67 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:39:52.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:52 smithi151 ceph-mon[129163]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T02:39:52.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:52 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1219499727' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T02:39:52.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:52 smithi151 ceph-mon[129163]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T02:39:52.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:52 smithi151 ceph-mon[129163]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T02:39:52.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:52 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3016380686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:52.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:52 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/3205447543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:39:53.240 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.238+0000 7fc46c8a9700 1 -- 172.21.15.17:0/2273283045 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc46009ad40 con 0x7fc468101250 2023-07-20T02:39:53.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc4500a1150 msgr2=0x7fc4500a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:53.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc4500a1150 0x7fc4500a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc468100710 tx=0x7fc458009260 comp rx=0 tx=0).stop 2023-07-20T02:39:53.242 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 msgr2=0x7fc4681021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:53.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc4681021d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc460007990 tx=0x7fc460008eb0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 shutdown_connections 2023-07-20T02:39:53.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc4500a1150 0x7fc4500a3610 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc468101250 0x7fc4681021d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4680ff6c0 0x7fc468103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 --2- 172.21.15.17:0/2273283045 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4681008b0 0x7fc468101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.243 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 >> 172.21.15.17:0/2273283045 conn(0x7fc4680fac70 msgr2=0x7fc468068b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:53.244 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 shutdown_connections 2023-07-20T02:39:53.244 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.241+0000 7fc46d8ab700 1 -- 172.21.15.17:0/2273283045 wait complete. 2023-07-20T02:39:53.255 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2023-07-20T02:39:53.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-07-20T02:39:53.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.384+0000 7f8034d43700 1 -- 172.21.15.17:0/2809238178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030100880 msgr2=0x7f8030100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:53.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.384+0000 7f8034d43700 1 --2- 172.21.15.17:0/2809238178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030100880 0x7f8030100ce0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f80240099c0 tx=0x7f8024009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.384+0000 7f8034d43700 1 -- 172.21.15.17:0/2809238178 shutdown_connections 2023-07-20T02:39:53.386 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.384+0000 7f8034d43700 1 --2- 172.21.15.17:0/2809238178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8030101220 0x7f8030107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.384+0000 7f8034d43700 1 --2- 172.21.15.17:0/2809238178 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80300ff690 0x7f80300ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.384+0000 7f8034d43700 1 --2- 172.21.15.17:0/2809238178 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030100880 0x7f8030100ce0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.384+0000 7f8034d43700 1 -- 172.21.15.17:0/2809238178 >> 172.21.15.17:0/2809238178 conn(0x7f80300fac00 msgr2=0x7f80300fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:53.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 -- 172.21.15.17:0/2809238178 shutdown_connections 2023-07-20T02:39:53.387 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 -- 172.21.15.17:0/2809238178 wait complete. 2023-07-20T02:39:53.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 Processor -- start 2023-07-20T02:39:53.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 -- start start 2023-07-20T02:39:53.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80300ff690 0x7f8030198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:53.389 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8030100880 0x7f8030198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:53.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030101220 0x7f803019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:53.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80301a01f0 con 0x7f8030101220 2023-07-20T02:39:53.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80301a0330 con 0x7f80300ff690 2023-07-20T02:39:53.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.385+0000 7f8034d43700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80301a04c0 con 0x7f8030100880 2023-07-20T02:39:53.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030101220 0x7f803019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:53.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030101220 0x7f803019fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43644/0 (socket says 172.21.15.17:43644) 2023-07-20T02:39:53.390 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 -- 172.21.15.17:0/999473437 learned_addr learned my addr 172.21.15.17:0/999473437 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:53.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802e59c700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80300ff690 0x7f8030198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:53.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802dd9b700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8030100880 0x7f8030198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:53.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 -- 172.21.15.17:0/999473437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8030100880 msgr2=0x7f8030198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:53.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8030100880 0x7f8030198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 -- 172.21.15.17:0/999473437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80300ff690 msgr2=0x7f8030198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:53.391 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80300ff690 0x7f8030198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:53.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 -- 172.21.15.17:0/999473437 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80240095f0 con 0x7f8030101220 2023-07-20T02:39:53.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802dd9b700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8030100880 0x7f8030198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:39:53.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f802ed9d700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030101220 0x7f803019fc20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8028007ee0 tx=0x7f802800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:53.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f801f7fe700 1 -- 172.21.15.17:0/999473437 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802800ddd0 con 0x7f8030101220 2023-07-20T02:39:53.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f801f7fe700 1 -- 172.21.15.17:0/999473437 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f802800f040 con 0x7f8030101220 2023-07-20T02:39:53.392 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80301a07a0 con 0x7f8030101220 2023-07-20T02:39:53.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f801f7fe700 1 -- 172.21.15.17:0/999473437 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80280048e0 con 0x7f8030101220 2023-07-20T02:39:53.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.386+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80301a0c60 con 0x7f8030101220 2023-07-20T02:39:53.393 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.387+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803010aea0 con 0x7f8030101220 2023-07-20T02:39:53.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.388+0000 7f801f7fe700 1 -- 172.21.15.17:0/999473437 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8028003a00 con 0x7f8030101220 2023-07-20T02:39:53.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.389+0000 7f801f7fe700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80180a1150 0x7f80180a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:53.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.389+0000 7f801f7fe700 1 -- 172.21.15.17:0/999473437 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f80280cc910 con 0x7f8030101220 2023-07-20T02:39:53.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.389+0000 7f802e59c700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80180a1150 0x7f80180a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:53.394 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.389+0000 7f802e59c700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80180a1150 0x7f80180a3610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f80301006e0 tx=0x7f802000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:53.395 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.392+0000 7f801f7fe700 1 -- 172.21.15.17:0/999473437 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f80280961b0 con 0x7f8030101220 2023-07-20T02:39:53.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.151:6838/4007685317,v1:172.21.15.151:6839/4007685317] up:active 2023-07-20T02:39:53.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:53.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T02:39:53.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:53.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2273283045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:39:53.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:53.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:53.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:53 smithi146 ceph-mon[128841]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.151:6838/4007685317,v1:172.21.15.151:6839/4007685317] up:active 2023-07-20T02:39:53.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:53.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T02:39:53.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:53.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2273283045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:39:53.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.614 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.615 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:53.615 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:53.615 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:53 smithi151 ceph-mon[129163]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.636 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:53.635+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f80301a1620 con 0x7f8030101220 2023-07-20T02:39:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.151:6838/4007685317,v1:172.21.15.151:6839/4007685317] up:active 2023-07-20T02:39:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T02:39:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2273283045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:39:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:53.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:39:53.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:39:53.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:53 smithi017 ceph-mon[127345]: from='mgr.14152 172.21.15.17:0/3491641352' entity='mgr.x' 2023-07-20T02:39:54.245 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.244+0000 7f801f7fe700 1 -- 172.21.15.17:0/999473437 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f80280230b0 con 0x7f8030101220 2023-07-20T02:39:54.248 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80180a1150 msgr2=0x7f80180a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:54.248 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80180a1150 0x7f80180a3610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f80301006e0 tx=0x7f802000b410 comp rx=0 tx=0).stop 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030101220 msgr2=0x7f803019fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030101220 0x7f803019fc20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8028007ee0 tx=0x7f802800d3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 shutdown_connections 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f80180a1150 0x7f80180a3610 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f8030101220 0x7f803019fc20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80300ff690 0x7f8030198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 --2- 172.21.15.17:0/999473437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8030100880 0x7f8030198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.249 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 >> 172.21.15.17:0/999473437 conn(0x7f80300fac00 msgr2=0x7f80300688c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:54.250 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 shutdown_connections 2023-07-20T02:39:54.250 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.247+0000 7f8034d43700 1 -- 172.21.15.17:0/999473437 wait complete. 2023-07-20T02:39:54.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-20T02:39:54.402 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.400+0000 7ff225da6700 1 -- 172.21.15.17:0/3819293374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff220100760 msgr2=0x7ff220100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:54.402 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.400+0000 7ff225da6700 1 --2- 172.21.15.17:0/3819293374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff220100760 0x7ff220100b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff2100099c0 tx=0x7ff210009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.402 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.400+0000 7ff225da6700 1 -- 172.21.15.17:0/3819293374 shutdown_connections 2023-07-20T02:39:54.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.400+0000 7ff225da6700 1 --2- 172.21.15.17:0/3819293374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2201022f0 0x7ff220108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.400+0000 7ff225da6700 1 --2- 172.21.15.17:0/3819293374 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff220100760 0x7ff220100b80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.400+0000 7ff225da6700 1 --2- 172.21.15.17:0/3819293374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff220101950 0x7ff220101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.400+0000 7ff225da6700 1 -- 172.21.15.17:0/3819293374 >> 172.21.15.17:0/3819293374 conn(0x7ff2200fbcf0 msgr2=0x7ff2200fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:54.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.401+0000 7ff225da6700 1 -- 172.21.15.17:0/3819293374 shutdown_connections 2023-07-20T02:39:54.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.401+0000 7ff225da6700 1 -- 172.21.15.17:0/3819293374 wait complete. 2023-07-20T02:39:54.403 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.401+0000 7ff225da6700 1 Processor -- start 2023-07-20T02:39:54.405 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.401+0000 7ff225da6700 1 -- start start 2023-07-20T02:39:54.405 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.401+0000 7ff225da6700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff220100760 0x7ff220196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:54.405 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff225da6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff220101950 0x7ff2201969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:54.405 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff225da6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2201022f0 0x7ff22019da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:54.406 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff225da6700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22019dfe0 con 0x7ff2201022f0 2023-07-20T02:39:54.406 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff225da6700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22019e120 con 0x7ff220100760 2023-07-20T02:39:54.406 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff225da6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22019e2b0 con 0x7ff220101950 2023-07-20T02:39:54.406 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2201022f0 0x7ff22019da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:54.406 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2201022f0 0x7ff22019da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43658/0 (socket says 172.21.15.17:43658) 2023-07-20T02:39:54.406 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 -- 172.21.15.17:0/706394113 learned_addr learned my addr 172.21.15.17:0/706394113 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21f7fe700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff220100760 0x7ff220196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21effd700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff220101950 0x7ff2201969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 -- 172.21.15.17:0/706394113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff220101950 msgr2=0x7ff2201969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff220101950 0x7ff2201969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 -- 172.21.15.17:0/706394113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff220100760 msgr2=0x7ff220196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff220100760 0x7ff220196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 -- 172.21.15.17:0/706394113 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2100095f0 con 0x7ff2201022f0 2023-07-20T02:39:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21ffff700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2201022f0 0x7ff22019da10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff218007ee0 tx=0x7ff21800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:54.408 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21cff9700 1 -- 172.21.15.17:0/706394113 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218017070 con 0x7ff2201022f0 2023-07-20T02:39:54.408 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21cff9700 1 -- 172.21.15.17:0/706394113 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff21800f040 con 0x7ff2201022f0 2023-07-20T02:39:54.408 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22019e590 con 0x7ff2201022f0 2023-07-20T02:39:54.408 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff21cff9700 1 -- 172.21.15.17:0/706394113 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218005740 con 0x7ff2201022f0 2023-07-20T02:39:54.408 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.402+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22019ea50 con 0x7ff2201022f0 2023-07-20T02:39:54.408 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.403+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22010bf70 con 0x7ff2201022f0 2023-07-20T02:39:54.409 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.404+0000 7ff21cff9700 1 -- 172.21.15.17:0/706394113 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff2180058a0 con 0x7ff2201022f0 2023-07-20T02:39:54.409 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.405+0000 7ff21cff9700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff2080a1150 0x7ff2080a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:54.410 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.405+0000 7ff21f7fe700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff2080a1150 0x7ff2080a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:54.410 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.405+0000 7ff21cff9700 1 -- 172.21.15.17:0/706394113 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff2180cc080 con 0x7ff2201022f0 2023-07-20T02:39:54.410 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.405+0000 7ff21f7fe700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff2080a1150 0x7ff2080a3610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff21000b580 tx=0x7ff210005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:54.410 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.409+0000 7ff21cff9700 1 -- 172.21.15.17:0/706394113 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff2180fe780 con 0x7ff2201022f0 2023-07-20T02:39:54.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:54 smithi146 ceph-mon[128841]: pgmap v257: 97 pgs: 29 creating+peering, 29 active+clean, 39 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:39:54.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:54 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/2273283045' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T02:39:54.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:54 smithi146 ceph-mon[128841]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:54.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:54 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/999473437' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-20T02:39:54.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:54 smithi151 ceph-mon[129163]: pgmap v257: 97 pgs: 29 creating+peering, 29 active+clean, 39 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:39:54.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:54 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/2273283045' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T02:39:54.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:54 smithi151 ceph-mon[129163]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:54.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:54 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/999473437' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-20T02:39:54.658 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:54.657+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7ff22010c100 con 0x7ff2201022f0 2023-07-20T02:39:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:54 smithi017 ceph-mon[127345]: pgmap v257: 97 pgs: 29 creating+peering, 29 active+clean, 39 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:39:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:54 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/2273283045' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T02:39:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:54 smithi017 ceph-mon[127345]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:54 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/999473437' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-20T02:39:55.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.031+0000 7ff21cff9700 1 -- 172.21.15.17:0/706394113 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff218095880 con 0x7ff2201022f0 2023-07-20T02:39:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.033+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff2080a1150 msgr2=0x7ff2080a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.033+0000 7ff225da6700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff2080a1150 0x7ff2080a3610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff21000b580 tx=0x7ff210005fb0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.033+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2201022f0 msgr2=0x7ff22019da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.033+0000 7ff225da6700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2201022f0 0x7ff22019da10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff218007ee0 tx=0x7ff21800d3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 shutdown_connections 2023-07-20T02:39:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7ff2080a1150 0x7ff2080a3610 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff2201022f0 0x7ff22019da10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.036 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff220100760 0x7ff220196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 --2- 172.21.15.17:0/706394113 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff220101950 0x7ff2201969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 >> 172.21.15.17:0/706394113 conn(0x7ff2200fbcf0 msgr2=0x7ff2200fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 shutdown_connections 2023-07-20T02:39:55.037 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.034+0000 7ff225da6700 1 -- 172.21.15.17:0/706394113 wait complete. 2023-07-20T02:39:55.052 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-20T02:39:55.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 -- 172.21.15.17:0/2568780055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0101980 msgr2=0x7f64a0101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:55.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 --2- 172.21.15.17:0/2568780055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0101980 0x7f64a0101de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64940099c0 tx=0x7f6494009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 -- 172.21.15.17:0/2568780055 shutdown_connections 2023-07-20T02:39:55.184 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 --2- 172.21.15.17:0/2568780055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a0102320 0x7f64a0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 --2- 172.21.15.17:0/2568780055 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a0100790 0x7f64a0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 --2- 172.21.15.17:0/2568780055 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0101980 0x7f64a0101de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 -- 172.21.15.17:0/2568780055 >> 172.21.15.17:0/2568780055 conn(0x7f64a00fbce0 msgr2=0x7f64a00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:55.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 -- 172.21.15.17:0/2568780055 shutdown_connections 2023-07-20T02:39:55.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.182+0000 7f64a685c700 1 -- 172.21.15.17:0/2568780055 wait complete. 2023-07-20T02:39:55.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 Processor -- start 2023-07-20T02:39:55.185 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 -- start start 2023-07-20T02:39:55.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a0100790 0x7f64a0196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:55.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a0101980 0x7f64a0196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:55.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0102320 0x7f64a019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:55.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a019e0a0 con 0x7f64a0102320 2023-07-20T02:39:55.187 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a019e1e0 con 0x7f64a0101980 2023-07-20T02:39:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a685c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a019e370 con 0x7f64a0100790 2023-07-20T02:39:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0102320 0x7f64a019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0102320 0x7f64a019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43672/0 (socket says 172.21.15.17:43672) 2023-07-20T02:39:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 -- 172.21.15.17:0/1181968456 learned_addr learned my addr 172.21.15.17:0/1181968456 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f649f7fe700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a0101980 0x7f64a0196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f649ffff700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a0100790 0x7f64a0196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 -- 172.21.15.17:0/1181968456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a0100790 msgr2=0x7f64a0196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a0100790 0x7f64a0196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 -- 172.21.15.17:0/1181968456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a0101980 msgr2=0x7f64a0196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a0101980 0x7f64a0196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.183+0000 7f64a4df9700 1 -- 172.21.15.17:0/1181968456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64940095f0 con 0x7f64a0102320 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.184+0000 7f64a4df9700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0102320 0x7f64a019dad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6498007ee0 tx=0x7f649800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.184+0000 7f649d7fa700 1 -- 172.21.15.17:0/1181968456 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498017070 con 0x7f64a0102320 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.184+0000 7f649d7fa700 1 -- 172.21.15.17:0/1181968456 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f649800f040 con 0x7f64a0102320 2023-07-20T02:39:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.184+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a019e650 con 0x7f64a0102320 2023-07-20T02:39:55.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.184+0000 7f649d7fa700 1 -- 172.21.15.17:0/1181968456 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498005740 con 0x7f64a0102320 2023-07-20T02:39:55.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.184+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a019eb10 con 0x7f64a0102320 2023-07-20T02:39:55.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.185+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0107d20 con 0x7f64a0102320 2023-07-20T02:39:55.190 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.186+0000 7f649d7fa700 1 -- 172.21.15.17:0/1181968456 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f64980058a0 con 0x7f64a0102320 2023-07-20T02:39:55.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.186+0000 7f649d7fa700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64880a1150 0x7f64880a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:55.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.187+0000 7f649ffff700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64880a1150 0x7f64880a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:55.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.187+0000 7f649d7fa700 1 -- 172.21.15.17:0/1181968456 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f64980cc710 con 0x7f64a0102320 2023-07-20T02:39:55.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.187+0000 7f649ffff700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64880a1150 0x7f64880a3610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6490006fd0 tx=0x7f6490008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:55.191 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.190+0000 7f649d7fa700 1 -- 172.21.15.17:0/1181968456 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6498095fb0 con 0x7f64a0102320 2023-07-20T02:39:55.363 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-i[155438]: 2023-07-20T02:39:55.023+0000 7fedcdb89700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-20T02:39:55.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T02:39:55.364 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/999473437' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-20T02:39:55.364 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T02:39:55.364 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:55.364 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T02:39:55.364 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T02:39:55.364 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/706394113' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:39:55.365 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.146:6838/1321727767,v1:172.21.15.146:6839/1321727767] up:active 2023-07-20T02:39:55.365 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/706394113' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T02:39:55.365 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T02:39:55.365 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T02:39:55.365 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: Cluster is now healthy 2023-07-20T02:39:55.365 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 2 up:standby-replay 8 up:standby 2023-07-20T02:39:55.365 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:creating} 2 up:standby-replay 7 up:standby 2023-07-20T02:39:55.366 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:55 smithi151 ceph-mon[129163]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T02:39:55.437 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:55.436+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f64a002c970 con 0x7f64a0102320 2023-07-20T02:39:55.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T02:39:55.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/999473437' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-20T02:39:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T02:39:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T02:39:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T02:39:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/706394113' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:39:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.146:6838/1321727767,v1:172.21.15.146:6839/1321727767] up:active 2023-07-20T02:39:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/706394113' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T02:39:55.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T02:39:55.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T02:39:55.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: Cluster is now healthy 2023-07-20T02:39:55.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 2 up:standby-replay 8 up:standby 2023-07-20T02:39:55.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:creating} 2 up:standby-replay 7 up:standby 2023-07-20T02:39:55.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:55 smithi017 ceph-mon[127345]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T02:39:55.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T02:39:55.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/999473437' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-20T02:39:55.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T02:39:55.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T02:39:55.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T02:39:55.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T02:39:55.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/706394113' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:39:55.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.146:6838/1321727767,v1:172.21.15.146:6839/1321727767] up:active 2023-07-20T02:39:55.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/706394113' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T02:39:55.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T02:39:55.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T02:39:55.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: Cluster is now healthy 2023-07-20T02:39:55.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 2 up:standby-replay 8 up:standby 2023-07-20T02:39:55.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:creating} 2 up:standby-replay 7 up:standby 2023-07-20T02:39:55.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:55 smithi146 ceph-mon[128841]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T02:39:56.039 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.038+0000 7f649d7fa700 1 -- 172.21.15.17:0/1181968456 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f64980995d0 con 0x7f64a0102320 2023-07-20T02:39:56.042 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64880a1150 msgr2=0x7f64880a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64880a1150 0x7f64880a3610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6490006fd0 tx=0x7f6490008040 comp rx=0 tx=0).stop 2023-07-20T02:39:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0102320 msgr2=0x7f64a019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.043 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0102320 0x7f64a019dad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6498007ee0 tx=0x7f649800d3b0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 shutdown_connections 2023-07-20T02:39:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f64880a1150 0x7f64880a3610 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f64a0102320 0x7f64a019dad0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64a0100790 0x7f64a0196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 --2- 172.21.15.17:0/1181968456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a0101980 0x7f64a0196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.045 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 >> 172.21.15.17:0/1181968456 conn(0x7f64a00fbce0 msgr2=0x7f64a0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 shutdown_connections 2023-07-20T02:39:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.041+0000 7f64a685c700 1 -- 172.21.15.17:0/1181968456 wait complete. 2023-07-20T02:39:56.058 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2023-07-20T02:39:56.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2023-07-20T02:39:56.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 -- 172.21.15.17:0/1894197848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c81011f0 msgr2=0x7f71c8107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/1894197848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c81011f0 0x7f71c8107aa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f71c000b0d0 tx=0x7f71c000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 -- 172.21.15.17:0/1894197848 shutdown_connections 2023-07-20T02:39:56.196 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/1894197848 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c81011f0 0x7f71c8107aa0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/1894197848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71c80ff660 0x7f71c80ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/1894197848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c8100850 0x7f71c8100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 -- 172.21.15.17:0/1894197848 >> 172.21.15.17:0/1894197848 conn(0x7f71c80fabd0 msgr2=0x7f71c80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:56.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 -- 172.21.15.17:0/1894197848 shutdown_connections 2023-07-20T02:39:56.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.194+0000 7f71cf1a4700 1 -- 172.21.15.17:0/1894197848 wait complete. 2023-07-20T02:39:56.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 Processor -- start 2023-07-20T02:39:56.197 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 -- start start 2023-07-20T02:39:56.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c80ff660 0x7f71c8195430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.199 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8100850 0x7f71c8195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71c81011f0 0x7f71c819c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c819cfb0 con 0x7f71c8100850 2023-07-20T02:39:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c819d0f0 con 0x7f71c80ff660 2023-07-20T02:39:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cf1a4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c819d280 con 0x7f71c81011f0 2023-07-20T02:39:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8100850 0x7f71c8195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8100850 0x7f71c8195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43684/0 (socket says 172.21.15.17:43684) 2023-07-20T02:39:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 -- 172.21.15.17:0/2088132803 learned_addr learned my addr 172.21.15.17:0/2088132803 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71ccf40700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c80ff660 0x7f71c8195430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 -- 172.21.15.17:0/2088132803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71c81011f0 msgr2=0x7f71c819c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71cd741700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71c81011f0 0x7f71c819c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71c81011f0 0x7f71c819c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 -- 172.21.15.17:0/2088132803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c80ff660 msgr2=0x7f71c8195430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c80ff660 0x7f71c8195430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.195+0000 7f71c7fff700 1 -- 172.21.15.17:0/2088132803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0009d00 con 0x7f71c8100850 2023-07-20T02:39:56.201 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71ccf40700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c80ff660 0x7f71c8195430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71cd741700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71c81011f0 0x7f71c819c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71c7fff700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8100850 0x7f71c8195970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f71b800d980 tx=0x7f71b800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71c5ffb700 1 -- 172.21.15.17:0/2088132803 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b800bb60 con 0x7f71c8100850 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71c5ffb700 1 -- 172.21.15.17:0/2088132803 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71b8007640 con 0x7f71c8100850 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c819d560 con 0x7f71c8100850 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71c5ffb700 1 -- 172.21.15.17:0/2088132803 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8004e90 con 0x7f71c8100850 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.196+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c819dac0 con 0x7f71c8100850 2023-07-20T02:39:56.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.197+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c810ae80 con 0x7f71c8100850 2023-07-20T02:39:56.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.198+0000 7f71c5ffb700 1 -- 172.21.15.17:0/2088132803 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f71b800bcc0 con 0x7f71c8100850 2023-07-20T02:39:56.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.198+0000 7f71c5ffb700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f71b40a11d0 0x7f71b40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.199+0000 7f71ccf40700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f71b40a11d0 0x7f71b40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.203 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.199+0000 7f71c5ffb700 1 -- 172.21.15.17:0/2088132803 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f71b80cc670 con 0x7f71c8100850 2023-07-20T02:39:56.204 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.199+0000 7f71ccf40700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f71b40a11d0 0x7f71b40a3690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71b0005950 tx=0x7f71b00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:56.204 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.203+0000 7f71c5ffb700 1 -- 172.21.15.17:0/2088132803 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f71b8095e70 con 0x7f71c8100850 2023-07-20T02:39:56.435 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.434+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f71c8060bb0 con 0x7f71b40a11d0 2023-07-20T02:39:56.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-mon[127345]: pgmap v258: 97 pgs: 20 creating+peering, 71 active+clean, 6 unknown; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2023-07-20T02:39:56.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1181968456' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T02:39:56.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-mon[127345]: mds.? [v2:172.21.15.146:6836/2995295414,v1:172.21.15.146:6837/2995295414] up:active 2023-07-20T02:39:56.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-mon[127345]: from='client.? 172.21.15.17:0/1181968456' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T02:39:56.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-mon[127345]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-07-20T02:39:56.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-mon[127345]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-20T02:39:56.462 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.461+0000 7f71c5ffb700 1 -- 172.21.15.17:0/2088132803 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f71c8060bb0 con 0x7f71b40a11d0 2023-07-20T02:39:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f71b40a11d0 msgr2=0x7f71b40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.464 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f71b40a11d0 0x7f71b40a3690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71b0005950 tx=0x7f71b00058e0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8100850 msgr2=0x7f71c8195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8100850 0x7f71c8195970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f71b800d980 tx=0x7f71b800dd40 comp rx=0 tx=0).stop 2023-07-20T02:39:56.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 shutdown_connections 2023-07-20T02:39:56.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7f71b40a11d0 0x7f71b40a3690 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71c81011f0 0x7f71c819c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c80ff660 0x7f71c8195430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 --2- 172.21.15.17:0/2088132803 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f71c8100850 0x7f71c8195970 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 >> 172.21.15.17:0/2088132803 conn(0x7f71c80fabd0 msgr2=0x7f71c80fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:56.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 shutdown_connections 2023-07-20T02:39:56.466 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.463+0000 7f71cf1a4700 1 -- 172.21.15.17:0/2088132803 wait complete. 2023-07-20T02:39:56.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs --group_name=qa sv_0 '' 2023-07-20T02:39:56.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:56 smithi146 ceph-mon[128841]: pgmap v258: 97 pgs: 20 creating+peering, 71 active+clean, 6 unknown; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2023-07-20T02:39:56.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:56 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1181968456' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T02:39:56.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:56 smithi146 ceph-mon[128841]: mds.? [v2:172.21.15.146:6836/2995295414,v1:172.21.15.146:6837/2995295414] up:active 2023-07-20T02:39:56.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:56 smithi146 ceph-mon[128841]: from='client.? 172.21.15.17:0/1181968456' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T02:39:56.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:56 smithi146 ceph-mon[128841]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-07-20T02:39:56.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:56 smithi146 ceph-mon[128841]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-20T02:39:56.609 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 -- 172.21.15.17:0/2638171528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280fe7a0 msgr2=0x7fc9280fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.609 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/2638171528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280fe7a0 0x7fc9280fec00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0099c0 tx=0x7fc91c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.609 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 -- 172.21.15.17:0/2638171528 shutdown_connections 2023-07-20T02:39:56.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/2638171528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280ff140 0x7fc928108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/2638171528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc928101d30 0x7fc928102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/2638171528 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280fe7a0 0x7fc9280fec00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 -- 172.21.15.17:0/2638171528 >> 172.21.15.17:0/2638171528 conn(0x7fc9280fa340 msgr2=0x7fc9280fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:56.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 -- 172.21.15.17:0/2638171528 shutdown_connections 2023-07-20T02:39:56.610 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.607+0000 7fc92c8a6700 1 -- 172.21.15.17:0/2638171528 wait complete. 2023-07-20T02:39:56.613 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 Processor -- start 2023-07-20T02:39:56.613 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 -- start start 2023-07-20T02:39:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280fe7a0 0x7fc928196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280ff140 0x7fc928196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc928101d30 0x7fc92819d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92819dfb0 con 0x7fc9280ff140 2023-07-20T02:39:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92819e0f0 con 0x7fc928101d30 2023-07-20T02:39:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc92c8a6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92819e280 con 0x7fc9280fe7a0 2023-07-20T02:39:56.614 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc925d9b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280ff140 0x7fc928196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc925d9b700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280ff140 0x7fc928196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43712/0 (socket says 172.21.15.17:43712) 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.608+0000 7fc925d9b700 1 -- 172.21.15.17:0/3376484847 learned_addr learned my addr 172.21.15.17:0/3376484847 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc926d9d700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc928101d30 0x7fc92819d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc92659c700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280fe7a0 0x7fc928196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc925d9b700 1 -- 172.21.15.17:0/3376484847 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280fe7a0 msgr2=0x7fc928196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc925d9b700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280fe7a0 0x7fc928196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc925d9b700 1 -- 172.21.15.17:0/3376484847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc928101d30 msgr2=0x7fc92819d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.615 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc925d9b700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc928101d30 0x7fc92819d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc925d9b700 1 -- 172.21.15.17:0/3376484847 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91c0095f0 con 0x7fc9280ff140 2023-07-20T02:39:56.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc925d9b700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280ff140 0x7fc928196970 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009fd0 tx=0x7fc91c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:56.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc9177fe700 1 -- 172.21.15.17:0/3376484847 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c01d070 con 0x7fc9280ff140 2023-07-20T02:39:56.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc9177fe700 1 -- 172.21.15.17:0/3376484847 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc91c0054f0 con 0x7fc9280ff140 2023-07-20T02:39:56.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92819e500 con 0x7fc9280ff140 2023-07-20T02:39:56.616 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc9177fe700 1 -- 172.21.15.17:0/3376484847 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c004c70 con 0x7fc9280ff140 2023-07-20T02:39:56.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.609+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92819e920 con 0x7fc9280ff140 2023-07-20T02:39:56.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.610+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc928107880 con 0x7fc9280ff140 2023-07-20T02:39:56.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.611+0000 7fc9177fe700 1 -- 172.21.15.17:0/3376484847 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc91c00fc30 con 0x7fc9280ff140 2023-07-20T02:39:56.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.612+0000 7fc9177fe700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc9100a1150 0x7fc9100a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:56.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.612+0000 7fc92659c700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc9100a1150 0x7fc9100a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:56.617 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.612+0000 7fc9177fe700 1 -- 172.21.15.17:0/3376484847 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc91c0cd710 con 0x7fc9280ff140 2023-07-20T02:39:56.618 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:56 smithi151 ceph-mon[129163]: pgmap v258: 97 pgs: 20 creating+peering, 71 active+clean, 6 unknown; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2023-07-20T02:39:56.618 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:56 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1181968456' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T02:39:56.618 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:56 smithi151 ceph-mon[129163]: mds.? [v2:172.21.15.146:6836/2995295414,v1:172.21.15.146:6837/2995295414] up:active 2023-07-20T02:39:56.618 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:56 smithi151 ceph-mon[129163]: from='client.? 172.21.15.17:0/1181968456' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T02:39:56.619 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:56 smithi151 ceph-mon[129163]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-07-20T02:39:56.619 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:56 smithi151 ceph-mon[129163]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-20T02:39:56.619 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.612+0000 7fc92659c700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc9100a1150 0x7fc9100a3610 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc9280675f0 tx=0x7fc918009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:56.619 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.616+0000 7fc9177fe700 1 -- 172.21.15.17:0/3376484847 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc91c097040 con 0x7fc9280ff140 2023-07-20T02:39:56.705 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:39:56.440+0000 7f85bec90700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:39:56.705 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:39:56.440+0000 7f85bec90700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:39:56.706 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:39:56.440+0000 7f85bec90700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:39:56.706 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:39:56.440+0000 7f85bec90700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:39:56.706 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:39:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:39:56.440+0000 7f85bec90700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:39:56.850 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.849+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc928060c40 con 0x7fc9100a1150 2023-07-20T02:39:56.876 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.875+0000 7fc9177fe700 1 -- 172.21.15.17:0/3376484847 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc928060c40 con 0x7fc9100a1150 2023-07-20T02:39:56.878 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc9100a1150 msgr2=0x7fc9100a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc9100a1150 0x7fc9100a3610 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc9280675f0 tx=0x7fc918009260 comp rx=0 tx=0).stop 2023-07-20T02:39:56.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280ff140 msgr2=0x7fc928196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:56.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280ff140 0x7fc928196970 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009fd0 tx=0x7fc91c00f740 comp rx=0 tx=0).stop 2023-07-20T02:39:56.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 shutdown_connections 2023-07-20T02:39:56.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fc9100a1150 0x7fc9100a3610 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.879 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc928101d30 0x7fc92819d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280fe7a0 0x7fc928196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 --2- 172.21.15.17:0/3376484847 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fc9280ff140 0x7fc928196970 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:56.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 >> 172.21.15.17:0/3376484847 conn(0x7fc9280fa340 msgr2=0x7fc9280fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:56.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 shutdown_connections 2023-07-20T02:39:56.880 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:56.877+0000 7fc92c8a6700 1 -- 172.21.15.17:0/3376484847 wait complete. 2023-07-20T02:39:56.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0 2023-07-20T02:39:57.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 -- 172.21.15.17:0/2569517948 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb808103b10 msgr2=0x7fb808105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:57.022 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 --2- 172.21.15.17:0/2569517948 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb808103b10 0x7fb808105f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009990 tx=0x7fb7f8009ca0 comp rx=0 tx=0).stop 2023-07-20T02:39:57.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 -- 172.21.15.17:0/2569517948 shutdown_connections 2023-07-20T02:39:57.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 --2- 172.21.15.17:0/2569517948 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb808106440 0x7fb8081088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:57.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 --2- 172.21.15.17:0/2569517948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8081011e0 0x7fb8081035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:57.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 --2- 172.21.15.17:0/2569517948 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb808103b10 0x7fb808105f00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:57.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 -- 172.21.15.17:0/2569517948 >> 172.21.15.17:0/2569517948 conn(0x7fb8080fabd0 msgr2=0x7fb8080fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:39:57.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 -- 172.21.15.17:0/2569517948 shutdown_connections 2023-07-20T02:39:57.023 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.021+0000 7fb80c808700 1 -- 172.21.15.17:0/2569517948 wait complete. 2023-07-20T02:39:57.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 Processor -- start 2023-07-20T02:39:57.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 -- start start 2023-07-20T02:39:57.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8081011e0 0x7fb808196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:57.026 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb808103b10 0x7fb808196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb808106440 0x7fb80819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 -- --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80819dfd0 con 0x7fb8081011e0 2023-07-20T02:39:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80819e110 con 0x7fb808103b10 2023-07-20T02:39:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80c808700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80819e2a0 con 0x7fb808106440 2023-07-20T02:39:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8081011e0 0x7fb808196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 --2- >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8081011e0 0x7fb808196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.17:3300/0 says I am v2:172.21.15.17:43726/0 (socket says 172.21.15.17:43726) 2023-07-20T02:39:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 -- 172.21.15.17:0/3163780266 learned_addr learned my addr 172.21.15.17:0/3163780266 (peer_addr_for_me v2:172.21.15.17:0/0) 2023-07-20T02:39:57.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb805d9b700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb808103b10 0x7fb808196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:57.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb806d9d700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb808106440 0x7fb80819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:57.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 -- 172.21.15.17:0/3163780266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb808106440 msgr2=0x7fb80819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:57.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb808106440 0x7fb80819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:57.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 -- 172.21.15.17:0/3163780266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb808103b10 msgr2=0x7fb808196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:39:57.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb808103b10 0x7fb808196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:39:57.028 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 -- 172.21.15.17:0/3163780266 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80095f0 con 0x7fb8081011e0 2023-07-20T02:39:57.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb806d9d700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb808106440 0x7fb80819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:39:57.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.022+0000 7fb80659c700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fb8081011e0 0x7fb808196450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00c920 tx=0x7fb7fc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:57.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.023+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mon.0 v2:172.21.15.17:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc007b00 con 0x7fb8081011e0 2023-07-20T02:39:57.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.023+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mon.0 v2:172.21.15.17:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7fc004cf0 con 0x7fb8081011e0 2023-07-20T02:39:57.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.023+0000 7fb80c808700 1 -- 172.21.15.17:0/3163780266 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80819e580 con 0x7fb8081011e0 2023-07-20T02:39:57.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.023+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mon.0 v2:172.21.15.17:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc005610 con 0x7fb8081011e0 2023-07-20T02:39:57.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.023+0000 7fb80c808700 1 -- 172.21.15.17:0/3163780266 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80819ea90 con 0x7fb8081011e0 2023-07-20T02:39:57.030 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.024+0000 7fb80c808700 1 -- 172.21.15.17:0/3163780266 --> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8080fc750 con 0x7fb8081011e0 2023-07-20T02:39:57.031 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.025+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mon.0 v2:172.21.15.17:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb7fc004750 con 0x7fb8081011e0 2023-07-20T02:39:57.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.025+0000 7fb7f77fe700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb7f00a1150 0x7fb7f00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:39:57.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.025+0000 7fb805d9b700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb7f00a1150 0x7fb7f00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:39:57.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.025+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mon.0 v2:172.21.15.17:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb7fc0cc7d0 con 0x7fb8081011e0 2023-07-20T02:39:57.032 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.026+0000 7fb805d9b700 1 --2- 172.21.15.17:0/3163780266 >> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] conn(0x7fb7f00a1150 0x7fb7f00a3610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8006010 tx=0x7fb7f800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:39:57.033 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.029+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mon.0 v2:172.21.15.17:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb7fc096070 con 0x7fb8081011e0 2023-07-20T02:39:57.266 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.265+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mon.0 v2:172.21.15.17:3300/0 7 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb7fc004a00 con 0x7fb8081011e0 2023-07-20T02:39:57.272 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.271+0000 7fb80c808700 1 -- 172.21.15.17:0/3163780266 --> [v2:172.21.15.17:6800/2503922589,v1:172.21.15.17:6801/2503922589] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb808060bb0 con 0x7fb7f00a1150 2023-07-20T02:39:57.274 INFO:teuthology.orchestra.run.smithi017.stderr:2023-07-20T02:39:57.273+0000 7fb7f77fe700 1 -- 172.21.15.17:0/3163780266 <== mgr.14152 v2:172.21.15.17:6800/2503922589 1 ==== mgr_command_reply(tid 0: -2 subvolume 'sv_0' does not exist) v1 ==== 39+0+0 (secure 0 0 0) 0x7fb808060bb0 con 0x7fb7f00a1150 2023-07-20T02:39:57.276 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENOENT: subvolume 'sv_0' does not exist 2023-07-20T02:39:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-20T02:39:57.280 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi017 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T02:39:57.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:39:57.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:39:57.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:39:57.382 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-20T02:39:57.383 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:39:57.400 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:39:57.419 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:39:57.440 INFO:tasks.cephadm:Stopping all daemons... 2023-07-20T02:39:57.441 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-20T02:39:57.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a 2023-07-20T02:39:57.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:57 smithi146 ceph-mon[128841]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:57.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:57 smithi146 ceph-mon[128841]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:57.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:57 smithi151 ceph-mon[129163]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:57.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:39:57 smithi151 ceph-mon[129163]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:57.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:57 smithi017 ceph-mon[127345]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:57.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:57 smithi017 ceph-mon[127345]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:39:57.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:57 smithi017 systemd[1]: Stopping Ceph mon.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:57.706 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:39:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:39:57.273+0000 7f85bec90700 -1 mgr.server reply reply (2) No such file or directory subvolume 'sv_0' does not exist 2023-07-20T02:39:58.038 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-a[127321]: 2023-07-20T02:39:57.769+0000 7ffb4cdee700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:39:58.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-a[127321]: 2023-07-20T02:39:57.769+0000 7ffb4cdee700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-20T02:39:58.425 INFO:journalctl@ceph.mon.a.smithi017.stdout:Jul 20 02:39:58 smithi017 bash[163826]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-a 2023-07-20T02:39:58.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.a.service' 2023-07-20T02:39:58.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:39:58.489 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-20T02:39:58.489 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-20T02:39:58.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.b 2023-07-20T02:39:58.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:58 smithi146 systemd[1]: Stopping Ceph mon.b for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:39:59.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-b[128817]: 2023-07-20T02:39:59.046+0000 7f68d7682700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:39:59.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-b[128817]: 2023-07-20T02:39:59.046+0000 7f68d7682700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-20T02:39:59.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jul 20 02:39:59 smithi146 bash[155356]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-b 2023-07-20T02:39:59.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.b.service' 2023-07-20T02:40:00.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:00.033 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-20T02:40:00.033 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-20T02:40:00.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.c 2023-07-20T02:40:00.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:40:00 smithi151 systemd[1]: Stopping Ceph mon.c for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:00.776 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:40:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-c[129134]: 2023-07-20T02:40:00.423+0000 7f9a1016a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:00.776 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:40:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-c[129134]: 2023-07-20T02:40:00.423+0000 7f9a1016a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-20T02:40:01.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Jul 20 02:40:00 smithi151 bash[156852]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mon-c 2023-07-20T02:40:01.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mon.c.service' 2023-07-20T02:40:01.243 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:01.244 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-20T02:40:01.245 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-20T02:40:01.245 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T02:40:01.617 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:40:01 smithi017 systemd[1]: Stopping Ceph mgr.x for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:01.955 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:40:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x[127589]: 2023-07-20T02:40:01.613+0000 7f86508c4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:01.956 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Jul 20 02:40:01 smithi017 bash[164349]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-x 2023-07-20T02:40:02.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.x.service' 2023-07-20T02:40:02.188 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:02.189 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-20T02:40:02.190 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-20T02:40:02.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.y 2023-07-20T02:40:02.569 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:40:02 smithi146 systemd[1]: Stopping Ceph mgr.y for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:03.186 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:40:02 smithi146 bash[155704]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-y 2023-07-20T02:40:03.187 INFO:journalctl@ceph.mgr.y.smithi146.stdout:Jul 20 02:40:02 smithi146 systemd[1]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-20T02:40:03.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.y.service' 2023-07-20T02:40:03.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:03.230 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-20T02:40:03.230 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-20T02:40:03.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.z 2023-07-20T02:40:03.613 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:40:03 smithi151 systemd[1]: Stopping Ceph mgr.z for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:03.941 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:40:03 smithi151 bash[157022]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mgr-z 2023-07-20T02:40:03.941 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Jul 20 02:40:03 smithi151 systemd[1]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-20T02:40:04.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mgr.z.service' 2023-07-20T02:40:04.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:04.165 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-20T02:40:04.165 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-20T02:40:04.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.0 2023-07-20T02:40:04.455 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:40:04 smithi017 systemd[1]: Stopping Ceph osd.0 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:04.955 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:40:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-0[140347]: 2023-07-20T02:40:04.512+0000 7fc49e5ce700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:04.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:40:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-0[140347]: 2023-07-20T02:40:04.512+0000 7fc49e5ce700 -1 osd.0 83 *** Got signal Terminated *** 2023-07-20T02:40:04.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:40:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-0[140347]: 2023-07-20T02:40:04.512+0000 7fc49e5ce700 -1 osd.0 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:10.455 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:40:10 smithi017 bash[164487]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-0 2023-07-20T02:40:11.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.0.service' 2023-07-20T02:40:11.414 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:40:11 smithi017 systemd[1]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.0.service: Succeeded. 2023-07-20T02:40:11.414 INFO:journalctl@ceph.osd.0.smithi017.stdout:Jul 20 02:40:11 smithi017 systemd[1]: Stopped Ceph osd.0 for 9d6085e8-26a5-11ee-9b34-001a4aab830c. 2023-07-20T02:40:11.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:11.430 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-20T02:40:11.430 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-20T02:40:11.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.1 2023-07-20T02:40:11.819 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:40:11 smithi146 systemd[1]: Stopping Ceph osd.1 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:12.320 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:40:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-1[133811]: 2023-07-20T02:40:11.820+0000 7fa8dfcda700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:12.320 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:40:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-1[133811]: 2023-07-20T02:40:11.820+0000 7fa8dfcda700 -1 osd.1 83 *** Got signal Terminated *** 2023-07-20T02:40:12.320 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:40:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-1[133811]: 2023-07-20T02:40:11.820+0000 7fa8dfcda700 -1 osd.1 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:18.070 INFO:journalctl@ceph.osd.1.smithi146.stdout:Jul 20 02:40:17 smithi146 bash[155847]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-1 2023-07-20T02:40:18.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.1.service' 2023-07-20T02:40:18.983 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:18.984 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-20T02:40:18.985 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-20T02:40:18.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.2 2023-07-20T02:40:19.340 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:40:19 smithi151 systemd[1]: Stopping Ceph osd.2 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:19.613 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:40:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-2[134964]: 2023-07-20T02:40:19.337+0000 7f12ae0dc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:19.614 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:40:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-2[134964]: 2023-07-20T02:40:19.337+0000 7f12ae0dc700 -1 osd.2 83 *** Got signal Terminated *** 2023-07-20T02:40:19.614 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:40:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-2[134964]: 2023-07-20T02:40:19.337+0000 7f12ae0dc700 -1 osd.2 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:25.114 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jul 20 02:40:24 smithi151 bash[157167]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-2 2023-07-20T02:40:26.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.2.service' 2023-07-20T02:40:26.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:26.428 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-20T02:40:26.429 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-20T02:40:26.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.3 2023-07-20T02:40:26.797 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:40:26 smithi017 systemd[1]: Stopping Ceph osd.3 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:27.206 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:40:26 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:40:26.793+0000 7f0504c73700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:27.206 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:40:26 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:40:26.793+0000 7f0504c73700 -1 osd.3 83 *** Got signal Terminated *** 2023-07-20T02:40:27.206 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:40:26 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:40:26.793+0000 7f0504c73700 -1 osd.3 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:30.456 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:40:30 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:40:30.194+0000 7f04ffa67700 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:07.817958+0000 front 2023-07-20T02:40:07.817992+0000 (oldest deadline 2023-07-20T02:40:30.118340+0000) 2023-07-20T02:40:30.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:30 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:30.400+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:31.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:31.329+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:31.648 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:40:31 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3[145714]: 2023-07-20T02:40:31.228+0000 7f04ffa67700 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:07.817958+0000 front 2023-07-20T02:40:07.817992+0000 (oldest deadline 2023-07-20T02:40:30.118340+0000) 2023-07-20T02:40:31.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:31 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:31.429+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:31.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:31 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:31.645+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:32.672 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:32 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:32.340+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:32.673 INFO:journalctl@ceph.osd.3.smithi017.stdout:Jul 20 02:40:32 smithi017 bash[164754]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-3 2023-07-20T02:40:32.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:32 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:32.466+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:32.864 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:32 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:32.368+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:32.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:32 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:32.669+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:33.614 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:33 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:33.240+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:33.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:33 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:33.320+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:33.649 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:33 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:33.340+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:33.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:33 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:33.460+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:33.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:33 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:33.645+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:34.051 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.3.service' 2023-07-20T02:40:34.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:34.092 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-20T02:40:34.093 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-20T02:40:34.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.4 2023-07-20T02:40:34.482 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:40:34 smithi146 systemd[1]: Stopping Ceph osd.4 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:34.614 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:34 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:34.222+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:34.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:34 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:34.356+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:34.638 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:34 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:34.319+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:34.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:34 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:34.479+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:34.820 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:40:34 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-4[139220]: 2023-07-20T02:40:34.651+0000 7f783bc4e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:34.821 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:40:34 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-4[139220]: 2023-07-20T02:40:34.651+0000 7f783bc4e700 -1 osd.4 83 *** Got signal Terminated *** 2023-07-20T02:40:34.821 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:40:34 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-4[139220]: 2023-07-20T02:40:34.651+0000 7f783bc4e700 -1 osd.4 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:34.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:34 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:34.635+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:35.114 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:34 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:34.618+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:35.579 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:35 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:35.236+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:35.579 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:35 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:35.347+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:35.626 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:35 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:35.295+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:35.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:35 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:35.512+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:35.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:35 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:35.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:35.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:35 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:35.623+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:36.589 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:36 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:36.286+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:36.596 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:36 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:36.232+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:36.597 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:36 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:36.298+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:36.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:36 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:36.522+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:36.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:36 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:36.592+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:36.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:36 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:36.587+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:37.575 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:37 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:37.248+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:37.576 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:37 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:37.272+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:37.603 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:37.252+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:37.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:37 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:37.542+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:37.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:37 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:37.571+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:37.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:37 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:37.599+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:38.554 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:38 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:38.225+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:38.555 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:38 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:38.235+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:38.607 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:38 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:38.302+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:38.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:38 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:38.523+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:38.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:38 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:38.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:38.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:38 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:38.604+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:38.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:38 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:38.604+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:39.559 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:39 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:39.213+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:39.560 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:39 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:39.258+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:39.560 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:39 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:39.258+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:39.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:39 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:39.327+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:39.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:39 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:39.327+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:39.707 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:39 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:39.559+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:39.707 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:39 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:39.559+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:39.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:39 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:39.494+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:39.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:39 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:39.556+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:39.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:39 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:39.556+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:40.433 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jul 20 02:40:40 smithi146 bash[156127]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-4 2023-07-20T02:40:40.433 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:40 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:40.215+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:40.515 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:40 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:40.256+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:40.516 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:40 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:40.256+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:40.516 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:40 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:40.222+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:40.516 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:40 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:40.222+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:40.591 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:40.284+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:40.591 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:40.284+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:40.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:40 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:40.480+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:40.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:40 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:40.480+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:40.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:40 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:40.511+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:40.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:40 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:40.511+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:40.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:40.588+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:40.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:40.588+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:41.320 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:41 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:41.226+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:41.520 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:41 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:41.177+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:41.520 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:41 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:41.177+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:41.521 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:41 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:41.305+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:41.521 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:41 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:41.305+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:41.640 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:41.308+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:41.640 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:41.308+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:41.687 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:41 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:41.451+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:41.687 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:41 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:41.451+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:41.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.4.service' 2023-07-20T02:40:41.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:41.737 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-20T02:40:41.737 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-20T02:40:41.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.5 2023-07-20T02:40:41.776 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:41 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:41.516+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:41.776 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:41 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:41.516+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:41.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:41.637+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:41.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:41 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:41.637+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:42.103 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:41 smithi151 systemd[1]: Stopping Ceph osd.5 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:42.364 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:42.269+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:42.364 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:42.269+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:42.364 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:42.099+0000 7f92601fc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:42.365 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:42.099+0000 7f92601fc700 -1 osd.5 83 *** Got signal Terminated *** 2023-07-20T02:40:42.365 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:42.099+0000 7f92601fc700 -1 osd.5 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:42.365 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:42.198+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:42.365 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:42.198+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:42.570 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:42 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:42.263+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:42.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:42 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:42.500+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:42.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:42 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:42.500+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:42.673 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:42 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:42.288+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:42.673 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:42 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:42.288+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:42.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:42.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:42.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:42 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:42.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:42.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:42 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:42.670+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:42.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:42 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:42.670+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:43.510 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:43 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:43.222+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:43.557 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:43 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:43.198+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:43.557 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:43 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:43.198+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:43.557 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:43 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:43.239+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:43.558 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:43 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:43.239+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:43.638 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:43.335+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:43.639 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:43.335+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:43.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:43 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:43.507+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:43.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:43 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:43.507+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:43.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:43 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:43.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:43.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:43 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:43.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:43.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:43.635+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:43.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:43 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:43.635+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:44.534 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:44 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:44.196+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:44.568 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:44.164+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:44.568 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:44.164+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:44.568 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:44.289+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:44.569 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:44.289+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:44.654 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:44 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:44.330+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:44.654 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:44 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:44.330+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:44.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:44 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:44.531+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:44.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:44 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:44.531+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:44.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:44.564+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:44.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:44 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:44.564+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:44.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:44 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:44.651+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:44.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:44 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:44.651+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:45.520 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:45 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:45.223+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:45.596 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:45.328+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:45.597 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:45.328+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:45.597 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:45.117+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:45.597 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:45.117+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:45.598 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:45.117+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.730531+0000 front 2023-07-20T02:40:23.730504+0000 (oldest deadline 2023-07-20T02:40:44.830774+0000) 2023-07-20T02:40:45.658 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:45 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:45.337+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:45.658 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:45 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:45.337+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:45.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:45 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:45.517+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:45.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:45 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:45.517+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:45.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:45.593+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:45.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:45 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:45.593+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:45.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:45 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:45.654+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:45.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:45 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:45.654+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:46.504 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:46.245+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:46.564 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:46.145+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.428702+0000 front 2023-07-20T02:40:08.428634+0000 (oldest deadline 2023-07-20T02:40:32.528771+0000) 2023-07-20T02:40:46.565 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:46.145+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:12.529445+0000 front 2023-07-20T02:40:12.529344+0000 (oldest deadline 2023-07-20T02:40:38.429472+0000) 2023-07-20T02:40:46.565 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5[140534]: 2023-07-20T02:40:46.145+0000 7f925a7ef700 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.730531+0000 front 2023-07-20T02:40:23.730504+0000 (oldest deadline 2023-07-20T02:40:44.830774+0000) 2023-07-20T02:40:46.565 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:46.376+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:46.565 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:46.376+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:46.672 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:46 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:46.297+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:46.673 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:46 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:46.297+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:46.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:46.501+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:46.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:46 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:46.501+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:46.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:46.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:46.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:46 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:46.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:46.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:46 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:46.670+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:46.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:46 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:46.670+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:47.508 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:47.225+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:47.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:47.358+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:47.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:47.358+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:47.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:47.581+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:47.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:47.581+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:47.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:47 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:47.581+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:47.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:47 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:47.265+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:47.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:47 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:47.265+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:47.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:47 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:47.265+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:47.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:47.506+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:47.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:47 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:47.506+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:48.114 INFO:journalctl@ceph.osd.5.smithi151.stdout:Jul 20 02:40:47 smithi151 bash[157437]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-5 2023-07-20T02:40:48.206 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:47 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:47.710+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:48.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:47 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:47.710+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:48.524 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:48 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:48.202+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:48.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:48.370+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:48.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:48.370+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:48.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:48.370+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:48.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:48.552+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:48.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:48.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:48.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:48 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:48.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:48.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:48 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:48.256+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:48.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:48 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:48.256+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:48.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:48 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:48.256+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:48.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:48 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:48.521+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:48.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:48 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:48.521+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:48.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:48 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:48.521+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:49.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.5.service' 2023-07-20T02:40:49.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:49.118 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-20T02:40:49.119 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-20T02:40:49.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.6 2023-07-20T02:40:49.150 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:48 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:48.709+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:49.150 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:48 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:48.709+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:49.457 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:49 smithi017 systemd[1]: Stopping Ceph osd.6 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:49.457 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:49.264+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:49.457 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:49.264+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:49.458 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:49.264+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:49.537 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:49 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:49.180+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:49.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:49.348+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:49.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:49.348+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:49.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:49.348+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:49.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:49.541+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:49.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:49.541+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:49.617 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:49 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:49.541+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:49.759 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:49.474+0000 7fafdbb39700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:49.759 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:49.474+0000 7fafdbb39700 -1 osd.6 83 *** Got signal Terminated *** 2023-07-20T02:40:49.760 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:49.474+0000 7fafdbb39700 -1 osd.6 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:49.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:49 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:49.534+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:49.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:49 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:49.534+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:49.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:49 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:49.534+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:50.206 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:49.756+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:50.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:49.756+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:50.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:49 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:49.756+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:50.541 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:50.229+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:50.541 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:50.229+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:50.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:50.327+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:50.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:50.327+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:50.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:50.327+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:50.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:50.542+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:50.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:50.542+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:50.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:50 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:50.542+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:50.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:50 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:50.240+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:50.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:50 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:50.240+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:50.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:50 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:50.240+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:50.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:50.538+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:50.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:50.538+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:50.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:50.538+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:51.206 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:50 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:50.769+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:51.206 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:50 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:50.769+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:51.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:50 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:50.769+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:51.556 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:51 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:51.213+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:51.556 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:51 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:51.213+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:51.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:51 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:51.352+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:51.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:51 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:51.352+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:51.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:51 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:51.353+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:51.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:51 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:51.545+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:51.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:51 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:51.545+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:51.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:51 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:51.545+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:51.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:51 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:51.283+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:51.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:51 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:51.283+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:51.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:51 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:51.283+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:51.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:51 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:51.553+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:51.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:51 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:51.553+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:51.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:51 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:51.553+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:52.206 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:51 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:51.736+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:52.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:51 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:51.736+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:52.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:51 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:51.736+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:52.513 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:52.236+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:52.513 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:52.236+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:52.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:52.357+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:52.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:52.357+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:52.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:52.357+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:52.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:52.577+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:52.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:52.577+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:52.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:52 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:52.577+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:52.693 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:52 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:52.275+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:52.694 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:52 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:52.275+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:52.694 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:52 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:52.275+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:52.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:52.510+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:52.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:52.510+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:52.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:52 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:52.510+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:52.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:52 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:52.690+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:52.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:52 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:52.690+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:52.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:52 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:52.690+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:53.570 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:53 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:53.234+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:53.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:53 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:53.234+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:53.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:53 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:53.466+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:53.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:53 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:53.466+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:53.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:53 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:53.466+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:53.576 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:53 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:53.316+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:53.577 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:53 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:53.316+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:53.577 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:53 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:53.316+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:53.706 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:53.289+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:53.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:53.289+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:53.707 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:53.289+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:53.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:53 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:53.572+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:53.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:53 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:53.572+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:53.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:53 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:53.572+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:54.206 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:53.704+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:54.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:53.704+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:54.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:53.704+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:54.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:53 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:53.704+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:40:54.570 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:54 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:54.233+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:54.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:54 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:54.233+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:54.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:54 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:54.233+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:40:54.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:54 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:54.422+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:54.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:54 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:54.422+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:54.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:54 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:54.422+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:54.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:54 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:54.422+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:40:54.610 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:54 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:54.291+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:54.610 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:54 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:54.291+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:54.611 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:54 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:54.291+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:54.669 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:54 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:54.301+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:09.009654+0000 front 2023-07-20T02:40:09.009859+0000 (oldest deadline 2023-07-20T02:40:31.910011+0000) 2023-07-20T02:40:54.669 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:54 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:54.301+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.311849+0000 front 2023-07-20T02:40:15.311954+0000 (oldest deadline 2023-07-20T02:40:38.811918+0000) 2023-07-20T02:40:54.670 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:54 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6[151268]: 2023-07-20T02:40:54.301+0000 7fafd8931700 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.613366+0000 front 2023-07-20T02:40:21.613533+0000 (oldest deadline 2023-07-20T02:40:46.313241+0000) 2023-07-20T02:40:54.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:54 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:54.607+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:54.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:54 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:54.607+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:54.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:54 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:54.607+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:54.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:54 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:54.665+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:54.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:54 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:54.665+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:54.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:54 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:54.665+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:54.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:54 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:54.665+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:40:55.457 INFO:journalctl@ceph.osd.6.smithi017.stdout:Jul 20 02:40:55 smithi017 bash[165012]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-6 2023-07-20T02:40:55.570 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:55 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:55.185+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:55.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:55 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:55.185+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:55.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:55 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:55.185+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:40:55.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:55 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:55.375+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:55.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:55 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:55.375+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:55.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:55 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:55.375+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:55.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:55 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:55.375+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:40:55.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:55.310+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:55.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:55.310+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:55.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:55.310+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:55.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:55.558+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:55.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:55.558+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:55.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:55 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:55.558+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:55.984 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:55 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:55.712+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:55.984 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:55 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:55.712+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:55.984 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:55 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:55.712+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:55.984 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:55 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:55.712+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:40:56.428 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:56.136+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:56.429 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:56.136+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:56.429 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:56.136+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:40:56.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:56.360+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:56.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:56.360+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:56.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:56.360+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:56.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:56.523+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:56.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:56.523+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:56.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:56 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:56.523+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:56.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.6.service' 2023-07-20T02:40:56.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:56.425+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:56.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:56.425+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:56.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:56.425+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:56.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:56.425+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:40:56.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:40:56.857 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-20T02:40:56.858 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-20T02:40:56.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.7 2023-07-20T02:40:57.151 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:56 smithi146 systemd[1]: Stopping Ceph osd.7 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:40:57.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:56.752+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:57.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:56.752+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:57.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:56.752+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:57.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:56 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:56.752+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:40:57.409 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:57.146+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:57.409 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:57.146+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:57.410 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:57.146+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:40:57.410 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:57.203+0000 7efeffca8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:40:57.410 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:57.203+0000 7efeffca8700 -1 osd.7 83 *** Got signal Terminated *** 2023-07-20T02:40:57.410 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:57.203+0000 7efeffca8700 -1 osd.7 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:40:57.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:57 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:57.351+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:57.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:57 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:57.351+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:57.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:57 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:57.351+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:57.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:57 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:57.495+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:57.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:57 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:57.495+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:57.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:57 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:57.495+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:57.820 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:57.404+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:57.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:57.405+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:57.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:57.405+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:57.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:57 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:57.405+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:40:58.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:57.707+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:58.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:57.708+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:58.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:57.708+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:58.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:57 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:57.708+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:40:58.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:58.183+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:58.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:58.183+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:58.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:58.183+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:40:58.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:58.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:58.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:58.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:58.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:58.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:58.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:58.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:40:58.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:58 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:58.317+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:58.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:58 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:58.317+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:58.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:58 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:58.317+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:58.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:58 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:58.317+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:40:58.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:58 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:58.531+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:58.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:58 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:58.531+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:58.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:58 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:58.531+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:58.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:58 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:58.697+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:58.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:58 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:58.697+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:58.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:58 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:58.697+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:58.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:58 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:58.697+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:40:59.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:59.219+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:40:59.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:59.219+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:40:59.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:40:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:40:59.219+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:40:59.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:59.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:40:59.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:59.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:40:59.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:59.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:40:59.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:40:59 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:40:59.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:40:59.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:59 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:59.344+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:40:59.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:59 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:59.344+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:40:59.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:59 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:59.344+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:40:59.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:40:59 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:40:59.344+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:40:59.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:59 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:59.534+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:40:59.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:59 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:59.534+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:40:59.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:40:59 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:40:59.534+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:40:59.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:59 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:59.692+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:40:59.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:59 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:59.692+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:40:59.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:59 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:59.692+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:40:59.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:40:59 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:40:59.692+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:00.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:00 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:41:00.227+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:41:00.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:00 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:41:00.227+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:41:00.572 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:00 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:41:00.227+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:41:00.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:00 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:00.383+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:00.573 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:00 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:00.383+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:00.573 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:00 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:00.383+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:00.573 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:00 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:00.383+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:00.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:00.359+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:00.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:00.359+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:00.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:00.359+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:00.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:00.359+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:00.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:00.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:00.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:00.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:00.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:00 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:00.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:00.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:00.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:00.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:00.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:00.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:00.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:00.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:00 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:00.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:01.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:01.368+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:01.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:01.368+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:01.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:01.368+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:01.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:01.368+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:01.571 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:41:01.270+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.456458+0000 front 2023-07-20T02:40:16.456485+0000 (oldest deadline 2023-07-20T02:40:39.956691+0000) 2023-07-20T02:41:01.572 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:41:01.270+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.457999+0000 front 2023-07-20T02:40:23.457968+0000 (oldest deadline 2023-07-20T02:40:49.358110+0000) 2023-07-20T02:41:01.572 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:41:01.270+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.358860+0000 front 2023-07-20T02:40:29.358739+0000 (oldest deadline 2023-07-20T02:40:53.959449+0000) 2023-07-20T02:41:01.572 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:01 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7[144783]: 2023-07-20T02:41:01.270+0000 7efef9a9a700 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:38.660419+0000 front 2023-07-20T02:40:38.660405+0000 (oldest deadline 2023-07-20T02:41:00.960671+0000) 2023-07-20T02:41:01.614 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:01.564+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:01.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:01.564+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:01.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:01.564+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:01.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:01.338+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:01.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:01.338+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:01.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:01.338+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:01.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:01 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:01.338+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:01.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:01.637+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:01.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:01.638+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:01.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:01.638+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:01.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:01 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:01.638+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:02.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:02.351+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:02.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:02.351+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:02.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:02.351+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:02.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:02.351+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:02.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:02.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:02.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:02.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:02.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:02 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:02.561+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:02.790 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:02 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:02.372+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:02.790 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:02 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:02.372+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:02.791 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:02 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:02.372+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:02.791 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:02 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:02.372+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:02.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:02.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:02.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:02.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:02.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:02.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:02.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:02 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:02.652+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:03.071 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jul 20 02:41:02 smithi146 bash[156393]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-7 2023-07-20T02:41:03.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:03.333+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:03.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:03.333+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:03.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:03.333+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:03.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:03.333+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:03.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:03.531+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:03.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:03.531+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:03.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:03.531+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:03.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:03 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:03.531+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:03.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:03 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:03.323+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:03.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:03 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:03.323+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:03.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:03 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:03.323+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:03.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:03 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:03.323+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:03.956 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:03 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:03.686+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:03.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:03 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:03.686+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:03.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:03 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:03.686+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:03.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:03 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:03.686+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:04.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.7.service' 2023-07-20T02:41:04.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:04.403 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-20T02:41:04.404 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-20T02:41:04.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.8 2023-07-20T02:41:04.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:04.556+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:04.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:04.556+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:04.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:04.556+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:04.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:04.556+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:04.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:04.341+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:04.617 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:04.341+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:04.617 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:04.341+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:04.617 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:04.341+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:04.618 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 systemd[1]: Stopping Ceph osd.8 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:04.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:04 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:04.344+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:04.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:04 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:04.344+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:04.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:04 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:04.344+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:04.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:04 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:04.344+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:04.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:04 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:04.344+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:05.115 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:04.754+0000 7f7ebc2ef700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:05.115 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:04.754+0000 7f7ebc2ef700 -1 osd.8 83 *** Got signal Terminated *** 2023-07-20T02:41:05.115 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:04 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:04.754+0000 7f7ebc2ef700 -1 osd.8 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:41:05.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:04.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:05.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:04.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:05.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:04.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:05.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:04.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:05.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:04 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:04.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:05.614 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:05.326+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:05.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:05.326+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:05.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:05.326+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:05.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:05.326+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:05.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:05.326+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.225286+0000 front 2023-07-20T02:40:39.225178+0000 (oldest deadline 2023-07-20T02:41:05.125234+0000) 2023-07-20T02:41:05.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:05.528+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:05.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:05.528+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:05.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:05.528+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:05.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:05 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:05.528+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:05.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:05 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:05.384+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:05.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:05 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:05.384+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:05.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:05 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:05.384+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:05.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:05 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:05.384+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:05.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:05 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:05.384+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:06.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:05 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:05.741+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:06.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:05 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:05.741+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:06.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:05 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:05.741+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:06.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:05 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:05.741+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:06.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:05 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:05.741+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:06.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:06.499+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:06.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:06.499+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:06.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:06.499+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:06.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:06.499+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:06.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:06.292+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:06.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:06.292+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:06.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:06.292+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:06.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:06.292+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:06.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:06 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:06.292+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.225286+0000 front 2023-07-20T02:40:39.225178+0000 (oldest deadline 2023-07-20T02:41:05.125234+0000) 2023-07-20T02:41:06.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:06 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:06.397+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:06.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:06 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:06.397+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:06.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:06 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:06.397+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:06.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:06 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:06.397+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:06.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:06 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:06.397+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:07.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:06 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:06.762+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:07.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:06 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:06.762+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:07.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:06 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:06.762+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:07.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:06 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:06.762+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:07.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:06 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:06.762+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:07.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:07.503+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:07.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:07.503+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:07.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:07.503+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:07.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:07.503+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:07.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:07.263+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:07.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:07.263+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:07.617 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:07.263+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:07.617 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:07.263+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:07.617 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:07 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:07.263+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.225286+0000 front 2023-07-20T02:40:39.225178+0000 (oldest deadline 2023-07-20T02:41:05.125234+0000) 2023-07-20T02:41:07.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:07 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:07.380+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:07.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:07 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:07.380+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:07.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:07 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:07.380+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:07.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:07 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:07.380+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:07.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:07 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:07.380+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:08.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:07 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:07.773+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:08.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:07 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:07.773+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:08.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:07 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:07.773+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:08.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:07 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:07.773+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:08.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:07 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:07.773+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:08.614 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:08.482+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:08.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:08.482+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:08.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:08.482+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:08.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:08.482+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:08.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:08.312+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:08.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:08.312+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:08.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:08.312+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:08.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:08.312+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:08.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:08.312+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.225286+0000 front 2023-07-20T02:40:39.225178+0000 (oldest deadline 2023-07-20T02:41:05.125234+0000) 2023-07-20T02:41:08.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:08 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:08.363+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:08.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:08 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:08.363+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:08.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:08 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:08.363+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:08.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:08 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:08.363+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:08.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:08 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:08.363+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:09.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:08 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:08.783+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:09.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:08 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:08.783+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:09.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:08 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:08.783+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:09.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:08 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:08.783+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:09.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:08 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:08.783+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:09.614 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:09.454+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:09.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:09.454+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:09.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:09.454+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:09.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:09.454+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:09.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:09.454+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:09.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:09.320+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.619501+0000 front 2023-07-20T02:40:06.619340+0000 (oldest deadline 2023-07-20T02:40:30.719475+0000) 2023-07-20T02:41:09.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:09.320+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.320940+0000 front 2023-07-20T02:40:15.320999+0000 (oldest deadline 2023-07-20T02:40:40.021130+0000) 2023-07-20T02:41:09.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:09.320+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:20.021526+0000 front 2023-07-20T02:40:20.021553+0000 (oldest deadline 2023-07-20T02:40:48.222392+0000) 2023-07-20T02:41:09.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:09.320+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:31.624342+0000 front 2023-07-20T02:40:31.624451+0000 (oldest deadline 2023-07-20T02:40:57.524210+0000) 2023-07-20T02:41:09.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:09.320+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.225286+0000 front 2023-07-20T02:40:39.225178+0000 (oldest deadline 2023-07-20T02:41:05.125234+0000) 2023-07-20T02:41:09.616 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:09 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8[145997]: 2023-07-20T02:41:09.320+0000 7f7eb70e3700 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:45.125739+0000 front 2023-07-20T02:40:45.125715+0000 (oldest deadline 2023-07-20T02:41:09.225893+0000) 2023-07-20T02:41:09.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:09 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:09.382+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:09.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:09 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:09.382+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:09.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:09 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:09.382+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:09.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:09 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:09.382+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:09.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:09 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:09.382+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:10.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:09 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:09.759+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:10.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:09 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:09.759+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:10.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:09 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:09.759+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:10.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:09 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:09.759+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:10.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:09 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:09.759+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:10.615 INFO:journalctl@ceph.osd.8.smithi151.stdout:Jul 20 02:41:10 smithi151 bash[157706]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-8 2023-07-20T02:41:10.615 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:10 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:10.407+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:10.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:10 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:10.407+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:10.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:10 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:10.407+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:10.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:10 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:10.407+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:10.616 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:10 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:10.407+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:10.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:10 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:10.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:10.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:10 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:10.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:10.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:10 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:10.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:10.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:10 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:10.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:10.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:10 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:10.389+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:11.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:10 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:10.721+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:11.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:10 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:10.721+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:11.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:10 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:10.721+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:11.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:10 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:10.721+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:11.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:10 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:10.721+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:11.795 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:11 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:11.433+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:11.795 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:11 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:11.433+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:11.796 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:11 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:11.433+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:11.796 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:11 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:11.433+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:11.796 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:11 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:11.433+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:11.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.8.service' 2023-07-20T02:41:11.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:11.402+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:11.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:11.402+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:11.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:11.402+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:11.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:11.402+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:11.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:11 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:11.402+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:11.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:11.844 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-20T02:41:11.844 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-20T02:41:11.845 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.9 2023-07-20T02:41:11.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:11.701+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:11.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:11.701+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:11.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:11.701+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:11.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:11.701+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:11.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:11 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:11.701+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:11.959 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:11 smithi017 systemd[1]: Stopping Ceph osd.9 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:12.707 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.221+0000 7fe8dfefd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:12.707 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.221+0000 7fe8dfefd700 -1 osd.9 83 *** Got signal Terminated *** 2023-07-20T02:41:12.707 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.221+0000 7fe8dfefd700 -1 osd.9 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:41:12.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:12 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:12.423+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:12.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:12 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:12.423+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:12.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:12 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:12.423+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:12.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:12 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:12.423+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:12.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:12 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:12.423+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:12.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:12 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:12.423+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:12.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:12 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:12.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:12.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:12 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:12.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:12.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:12 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:12.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:12.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:12 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:12.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:12.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:12 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:12.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:13.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:13.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:13.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:13.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:13.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:13.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:12 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:12.718+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.757074+0000 front 2023-07-20T02:40:43.757180+0000 (oldest deadline 2023-07-20T02:41:11.957722+0000) 2023-07-20T02:41:13.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:13 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:13.377+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:13.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:13 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:13.377+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:13.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:13 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:13.377+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:13.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:13 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:13.377+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:13.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:13 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:13.377+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:13.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:13 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:13.377+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:13.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:13 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:13.457+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:13.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:13 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:13.457+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:13.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:13 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:13.457+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:13.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:13 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:13.457+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:13.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:13 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:13.457+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:14.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:13.766+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:14.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:13.766+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:14.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:13.766+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:14.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:13.766+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:14.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:13.766+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:14.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:13 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:13.766+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.757074+0000 front 2023-07-20T02:40:43.757180+0000 (oldest deadline 2023-07-20T02:41:11.957722+0000) 2023-07-20T02:41:14.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:14 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:14.396+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:14.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:14 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:14.396+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:14.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:14 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:14.396+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:14.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:14 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:14.396+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:14.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:14 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:14.396+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:14.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:14 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:14.396+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:14.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:14 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:14.445+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:14.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:14 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:14.445+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:14.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:14 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:14.445+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:14.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:14 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:14.445+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:14.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:14 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:14.445+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:15.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:14.726+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:15.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:14.726+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:15.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:14.726+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:15.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:14.726+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:15.209 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:14.726+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:15.209 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:14 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:14.726+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.757074+0000 front 2023-07-20T02:40:43.757180+0000 (oldest deadline 2023-07-20T02:41:11.957722+0000) 2023-07-20T02:41:15.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:15 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:15.381+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:15.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:15 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:15.381+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:15.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:15 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:15.381+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:15.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:15 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:15.381+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:15.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:15 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:15.381+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:15.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:15 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:15.381+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:15.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:15 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:15.475+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:15.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:15 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:15.475+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:15.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:15 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:15.475+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:15.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:15 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:15.475+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:15.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:15 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:15.475+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:15.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:15.699+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:15.957 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:15.699+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:15.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:15.699+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:15.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:15.699+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:15.958 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:15.699+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:15.959 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:15 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:15.699+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.757074+0000 front 2023-07-20T02:40:43.757180+0000 (oldest deadline 2023-07-20T02:41:11.957722+0000) 2023-07-20T02:41:16.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:16 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:16.407+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:16.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:16 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:16.407+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:16.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:16 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:16.407+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:16.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:16 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:16.407+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:16.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:16 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:16.407+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:16.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:16 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:16.407+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:16.864 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:16 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:16.437+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:16.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:16 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:16.437+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:16.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:16 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:16.437+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:16.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:16 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:16.437+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:16.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:16 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:16.437+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:17.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:16.707+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:05.751443+0000 front 2023-07-20T02:40:05.751596+0000 (oldest deadline 2023-07-20T02:40:31.051522+0000) 2023-07-20T02:41:17.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:16.707+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:16.352800+0000 front 2023-07-20T02:40:16.352704+0000 (oldest deadline 2023-07-20T02:40:38.052678+0000) 2023-07-20T02:41:17.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:16.707+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:23.854638+0000 front 2023-07-20T02:40:23.854505+0000 (oldest deadline 2023-07-20T02:40:49.754581+0000) 2023-07-20T02:41:17.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:16.707+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.755005+0000 front 2023-07-20T02:40:29.755237+0000 (oldest deadline 2023-07-20T02:40:53.255212+0000) 2023-07-20T02:41:17.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:16.707+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.056729+0000 front 2023-07-20T02:40:39.056844+0000 (oldest deadline 2023-07-20T02:41:03.756715+0000) 2023-07-20T02:41:17.208 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:16 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9[156748]: 2023-07-20T02:41:16.707+0000 7fe8dacf1700 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.757074+0000 front 2023-07-20T02:40:43.757180+0000 (oldest deadline 2023-07-20T02:41:11.957722+0000) 2023-07-20T02:41:17.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:17 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:17.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:17.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:17 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:17.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:17.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:17 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:17.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:17.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:17 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:17.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:17.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:17 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:17.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:17.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:17 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:17.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:17.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:17 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:17.398+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:17.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:17 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:17.423+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:17.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:17 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:17.423+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:17.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:17 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:17.423+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:17.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:17 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:17.423+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:17.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:17 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:17.423+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:17.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:17 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:17.423+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:18.207 INFO:journalctl@ceph.osd.9.smithi017.stdout:Jul 20 02:41:17 smithi017 bash[165289]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-9 2023-07-20T02:41:18.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:18 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:18.364+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:18.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:18 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:18.364+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:18.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:18 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:18.364+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:18.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:18 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:18.364+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:18.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:18 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:18.364+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:18.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:18 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:18.364+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:18.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:18 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:18.364+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:18.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:18 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:18.434+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:18.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:18 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:18.434+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:18.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:18 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:18.434+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:18.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:18 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:18.434+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:18.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:18 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:18.435+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:18.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:18 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:18.435+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:19.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.9.service' 2023-07-20T02:41:19.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:19.318 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-20T02:41:19.318 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-20T02:41:19.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.10 2023-07-20T02:41:19.671 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.331+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:19.671 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.331+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:19.671 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.331+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:19.671 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.331+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:19.671 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.332+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:19.672 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.332+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:19.672 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.332+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:19.672 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 systemd[1]: Stopping Ceph osd.10 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:19.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:19.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:19.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:19.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:19.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:19.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:19.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:19.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:19.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:19.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:19.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:19 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:19.417+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:20.071 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.666+0000 7f848e390700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:20.071 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.666+0000 7f848e390700 -1 osd.10 83 *** Got signal Terminated *** 2023-07-20T02:41:20.071 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:19 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:19.666+0000 7f848e390700 -1 osd.10 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:41:20.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:20 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:20.295+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:20.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:20 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:20.295+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:20.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:20 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:20.295+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:20.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:20 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:20.295+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:20.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:20 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:20.295+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:20.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:20 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:20.295+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:20.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:20 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:20.295+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:20.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:20 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:20.448+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:20.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:20 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:20.448+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:20.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:20 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:20.448+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:20.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:20 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:20.448+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:20.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:20 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:20.448+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:20.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:20 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:20.448+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:21.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:21 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:21.301+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:21.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:21 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:21.301+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:21.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:21 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:21.301+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:21.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:21 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:21.301+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:21.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:21 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:21.301+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:21.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:21 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:21.301+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:21.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:21 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:21.301+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:21.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:21 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:21.453+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:21.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:21 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:21.453+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:21.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:21 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:21.453+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:21.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:21 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:21.453+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:21.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:21 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:21.453+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:21.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:21 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:21.453+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:22.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:22 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:22.282+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:22.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:22 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:22.282+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:22.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:22 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:22.282+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:22.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:22 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:22.282+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:22.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:22 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:22.282+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:22.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:22 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:22.282+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:22.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:22 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:22.282+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:22.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:22 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:22.493+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:22.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:22 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:22.493+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:22.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:22 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:22.493+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:22.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:22 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:22.493+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:22.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:22 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:22.493+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:22.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:22 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:22.493+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:23.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:23.317+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:23.821 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:23.317+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:23.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:23.317+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:23.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:23.317+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:23.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:23.317+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:23.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:23.317+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:23.822 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:23 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:23.317+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:23.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:23.496+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:23.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:23.496+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:23.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:23.496+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:23.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:23.496+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:23.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:23.496+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:23.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:23.496+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:24.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:24 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:24.303+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:06.957336+0000 front 2023-07-20T02:40:06.957253+0000 (oldest deadline 2023-07-20T02:40:29.857297+0000) 2023-07-20T02:41:24.571 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:24 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:24.303+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:14.458734+0000 front 2023-07-20T02:40:14.458676+0000 (oldest deadline 2023-07-20T02:40:39.758883+0000) 2023-07-20T02:41:24.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:24 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:24.303+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.960591+0000 front 2023-07-20T02:40:21.960560+0000 (oldest deadline 2023-07-20T02:40:47.860630+0000) 2023-07-20T02:41:24.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:24 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:24.303+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6814 osd.3 since back 2023-07-20T02:40:29.561558+0000 front 2023-07-20T02:40:29.561588+0000 (oldest deadline 2023-07-20T02:40:53.661676+0000) 2023-07-20T02:41:24.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:24 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:24.303+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.562712+0000 front 2023-07-20T02:40:39.562733+0000 (oldest deadline 2023-07-20T02:41:03.662840+0000) 2023-07-20T02:41:24.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:24 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:24.303+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.663686+0000 front 2023-07-20T02:40:43.663498+0000 (oldest deadline 2023-07-20T02:41:11.863859+0000) 2023-07-20T02:41:24.572 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:24 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10[150261]: 2023-07-20T02:41:24.303+0000 7f8489985700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:51.864542+0000 front 2023-07-20T02:40:51.864510+0000 (oldest deadline 2023-07-20T02:41:16.564494+0000) 2023-07-20T02:41:24.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:24 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:24.512+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:24.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:24 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:24.512+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:24.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:24 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:24.512+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:24.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:24 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:24.512+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:24.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:24 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:24.512+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:24.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:24 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:24.512+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:24.867 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:24 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:24.512+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:25.322 INFO:journalctl@ceph.osd.10.smithi146.stdout:Jul 20 02:41:25 smithi146 bash[156662]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-10 2023-07-20T02:41:25.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:25 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:25.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:25.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:25 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:25.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:25.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:25 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:25.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:25.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:25 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:25.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:25.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:25 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:25.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:25.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:25 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:25.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:25.867 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:25 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:25.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:26.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.10.service' 2023-07-20T02:41:26.659 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:26.660 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-20T02:41:26.661 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-20T02:41:26.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.11 2023-07-20T02:41:26.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:26.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:26.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:26.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:26.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:26.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:26.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:26.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:26.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:26.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:26.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:26.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:26.867 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:26.575+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:26.867 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:26 smithi151 systemd[1]: Stopping Ceph osd.11 for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:27.365 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.018+0000 7f61c7e24700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:27.365 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.018+0000 7f61c7e24700 -1 osd.11 83 *** Got signal Terminated *** 2023-07-20T02:41:27.365 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.018+0000 7f61c7e24700 -1 osd.11 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:41:27.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:27.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:27.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:27.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:27.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:27.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:27.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:27 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:27.553+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:28.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:28 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:28.515+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:28.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:28 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:28.515+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:28.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:28 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:28.515+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:28.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:28 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:28.515+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:28.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:28 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:28.515+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:28.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:28 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:28.515+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:28.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:28 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:28.515+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:29.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:29 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:29.559+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:29.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:29 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:29.559+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:29.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:29 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:29.559+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:29.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:29 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:29.559+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:29.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:29 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:29.559+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:29.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:29 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:29.559+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:29.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:29 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:29.559+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:30.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:30 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:30.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:30.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:30 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:30.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:30.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:30 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:30.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:30.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:30 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:30.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:30.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:30 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:30.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:30.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:30 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:30.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:30.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:30 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:30.550+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:31.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2023-07-20T02:40:08.987152+0000 front 2023-07-20T02:40:08.987122+0000 (oldest deadline 2023-07-20T02:40:33.687353+0000) 2023-07-20T02:41:31.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6804 osd.1 since back 2023-07-20T02:40:15.388590+0000 front 2023-07-20T02:40:15.388645+0000 (oldest deadline 2023-07-20T02:40:38.888549+0000) 2023-07-20T02:41:31.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6804 osd.2 since back 2023-07-20T02:40:21.189462+0000 front 2023-07-20T02:40:21.189563+0000 (oldest deadline 2023-07-20T02:40:47.089557+0000) 2023-07-20T02:41:31.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2023-07-20T02:40:39.391916+0000 front 2023-07-20T02:40:39.391944+0000 (oldest deadline 2023-07-20T02:41:03.492093+0000) 2023-07-20T02:41:31.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6812 osd.5 since back 2023-07-20T02:40:43.492507+0000 front 2023-07-20T02:40:43.492623+0000 (oldest deadline 2023-07-20T02:41:09.392616+0000) 2023-07-20T02:41:31.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.17:6822 osd.6 since back 2023-07-20T02:40:54.394891+0000 front 2023-07-20T02:40:54.394766+0000 (oldest deadline 2023-07-20T02:41:16.694746+0000) 2023-07-20T02:41:31.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.146:6820 osd.7 since back 2023-07-20T02:41:00.795995+0000 front 2023-07-20T02:41:00.796016+0000 (oldest deadline 2023-07-20T02:41:23.696007+0000) 2023-07-20T02:41:31.866 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:31 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11[151495]: 2023-07-20T02:41:31.509+0000 7f61c3419700 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.151:6820 osd.8 since back 2023-07-20T02:41:06.398086+0000 front 2023-07-20T02:41:06.398126+0000 (oldest deadline 2023-07-20T02:41:31.098162+0000) 2023-07-20T02:41:32.865 INFO:journalctl@ceph.osd.11.smithi151.stdout:Jul 20 02:41:32 smithi151 bash[157978]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-osd-11 2023-07-20T02:41:34.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@osd.11.service' 2023-07-20T02:41:34.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:34.066 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-20T02:41:34.067 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-20T02:41:34.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.a 2023-07-20T02:41:34.439 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:41:34 smithi017 systemd[1]: Stopping Ceph mds.a for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:34.707 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:41:34 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-a[162013]: 2023-07-20T02:41:34.435+0000 7f6712b81700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:34.708 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:41:34 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-a[162013]: 2023-07-20T02:41:34.435+0000 7f6712b81700 -1 mds.a *** got signal Terminated *** 2023-07-20T02:41:36.197 INFO:journalctl@ceph.mds.a.smithi017.stdout:Jul 20 02:41:35 smithi017 bash[165557]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-a 2023-07-20T02:41:36.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.a.service' 2023-07-20T02:41:36.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:36.240 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-20T02:41:36.240 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-20T02:41:36.241 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.d 2023-07-20T02:41:36.457 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:41:36 smithi017 systemd[1]: Stopping Ceph mds.d for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:36.957 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:41:36 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-d[161040]: 2023-07-20T02:41:36.608+0000 7f1f719c8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:36.958 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:41:36 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-d[161040]: 2023-07-20T02:41:36.608+0000 7f1f719c8700 -1 mds.d *** got signal Terminated *** 2023-07-20T02:41:39.957 INFO:journalctl@ceph.mds.d.smithi017.stdout:Jul 20 02:41:39 smithi017 bash[165703]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-d 2023-07-20T02:41:40.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.d.service' 2023-07-20T02:41:40.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:40.149 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-20T02:41:40.149 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-20T02:41:40.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.g 2023-07-20T02:41:40.457 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:41:40 smithi017 systemd[1]: Stopping Ceph mds.g for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:40.957 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:41:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-g[161629]: 2023-07-20T02:41:40.516+0000 7f2bf5f9b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:40.958 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:41:40 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-g[161629]: 2023-07-20T02:41:40.516+0000 7f2bf5f9b700 -1 mds.g *** got signal Terminated *** 2023-07-20T02:41:44.957 INFO:journalctl@ceph.mds.g.smithi017.stdout:Jul 20 02:41:44 smithi017 bash[165850]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-g 2023-07-20T02:41:44.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.g.service' 2023-07-20T02:41:45.014 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:45.014 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-20T02:41:45.015 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-20T02:41:45.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.j 2023-07-20T02:41:45.388 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:41:45 smithi017 systemd[1]: Stopping Ceph mds.j for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:45.707 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:41:45 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-j[160677]: 2023-07-20T02:41:45.383+0000 7f808a980700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:45.708 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:41:45 smithi017 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-j[160677]: 2023-07-20T02:41:45.383+0000 7f808a980700 -1 mds.j *** got signal Terminated *** 2023-07-20T02:41:49.708 INFO:journalctl@ceph.mds.j.smithi017.stdout:Jul 20 02:41:49 smithi017 bash[166002]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-j 2023-07-20T02:41:49.823 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.j.service' 2023-07-20T02:41:49.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:49.859 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-20T02:41:49.860 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-20T02:41:49.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.b 2023-07-20T02:41:50.283 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:41:49 smithi146 systemd[1]: Stopping Ceph mds.b for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:50.572 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:41:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-b[154153]: 2023-07-20T02:41:50.279+0000 7ffb84fea700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:50.572 INFO:journalctl@ceph.mds.b.smithi146.stdout:Jul 20 02:41:50 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-b[154153]: 2023-07-20T02:41:50.279+0000 7ffb84fea700 -1 mds.b *** got signal Terminated *** 2023-07-20T02:41:55.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.b.service' 2023-07-20T02:41:55.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:55.621 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-20T02:41:55.621 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-20T02:41:55.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.e 2023-07-20T02:41:56.041 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:41:55 smithi146 systemd[1]: Stopping Ceph mds.e for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:56.322 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:41:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-e[153472]: 2023-07-20T02:41:56.036+0000 7f208f6f5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:56.322 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:41:56 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-e[153472]: 2023-07-20T02:41:56.036+0000 7f208f6f5700 -1 mds.e *** got signal Terminated *** 2023-07-20T02:41:58.550 INFO:journalctl@ceph.mds.e.smithi146.stdout:Jul 20 02:41:58 smithi146 bash[157078]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-e 2023-07-20T02:41:58.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.e.service' 2023-07-20T02:41:58.589 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:41:58.589 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-20T02:41:58.590 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-20T02:41:58.590 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.h 2023-07-20T02:41:58.822 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:41:58 smithi146 systemd[1]: Stopping Ceph mds.h for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:41:59.322 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:41:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-h[153811]: 2023-07-20T02:41:58.939+0000 7ff97c7ae700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:41:59.322 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:41:58 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-h[153811]: 2023-07-20T02:41:58.939+0000 7ff97c7ae700 -1 mds.h *** got signal Terminated *** 2023-07-20T02:42:04.548 INFO:journalctl@ceph.mds.h.smithi146.stdout:Jul 20 02:42:04 smithi146 bash[157225]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-h 2023-07-20T02:42:04.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.h.service' 2023-07-20T02:42:04.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:42:04.775 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-20T02:42:04.776 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-20T02:42:04.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.k 2023-07-20T02:42:05.072 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:42:04 smithi146 systemd[1]: Stopping Ceph mds.k for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:42:05.572 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:42:05 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-k[153129]: 2023-07-20T02:42:05.150+0000 7f1df47ea700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:42:05.573 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:42:05 smithi146 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-k[153129]: 2023-07-20T02:42:05.150+0000 7f1df47ea700 -1 mds.k *** got signal Terminated *** 2023-07-20T02:42:07.572 INFO:journalctl@ceph.mds.k.smithi146.stdout:Jul 20 02:42:07 smithi146 bash[157363]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-k 2023-07-20T02:42:07.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.k.service' 2023-07-20T02:42:07.697 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:42:07.698 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-20T02:42:07.698 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-20T02:42:07.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.c 2023-07-20T02:42:08.059 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:42:07 smithi151 systemd[1]: Stopping Ceph mds.c for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:42:08.366 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:42:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-c[155101]: 2023-07-20T02:42:08.053+0000 7f7f2cddf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:42:08.366 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:42:08 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-c[155101]: 2023-07-20T02:42:08.053+0000 7f7f2cddf700 -1 mds.c *** got signal Terminated *** 2023-07-20T02:42:10.922 INFO:journalctl@ceph.mds.c.smithi151.stdout:Jul 20 02:42:10 smithi151 bash[158245]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-c 2023-07-20T02:42:11.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.c.service' 2023-07-20T02:42:11.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:42:11.146 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-20T02:42:11.147 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-20T02:42:11.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.f 2023-07-20T02:42:11.366 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:42:11 smithi151 systemd[1]: Stopping Ceph mds.f for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:42:11.866 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:42:11 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-f[154761]: 2023-07-20T02:42:11.518+0000 7f30a7292700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:42:11.866 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:42:11 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-f[154761]: 2023-07-20T02:42:11.518+0000 7f30a7292700 -1 mds.f *** got signal Terminated *** 2023-07-20T02:42:16.344 INFO:journalctl@ceph.mds.f.smithi151.stdout:Jul 20 02:42:16 smithi151 bash[158384]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-f 2023-07-20T02:42:16.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.f.service' 2023-07-20T02:42:16.383 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:42:16.384 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-20T02:42:16.384 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-20T02:42:16.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.i 2023-07-20T02:42:16.616 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:42:16 smithi151 systemd[1]: Stopping Ceph mds.i for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:42:17.116 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:42:16 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-i[155438]: 2023-07-20T02:42:16.758+0000 7fedd1b91700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:42:17.117 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:42:16 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-i[155438]: 2023-07-20T02:42:16.758+0000 7fedd1b91700 -1 mds.i *** got signal Terminated *** 2023-07-20T02:42:22.745 INFO:journalctl@ceph.mds.i.smithi151.stdout:Jul 20 02:42:22 smithi151 bash[158527]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-i 2023-07-20T02:42:22.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.i.service' 2023-07-20T02:42:22.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:42:22.971 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-20T02:42:22.972 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-20T02:42:22.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.l 2023-07-20T02:42:23.340 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:42:23 smithi151 systemd[1]: Stopping Ceph mds.l for 9d6085e8-26a5-11ee-9b34-001a4aab830c... 2023-07-20T02:42:23.616 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:42:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-l[154417]: 2023-07-20T02:42:23.334+0000 7f811a802700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:42:23.616 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:42:23 smithi151 ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-l[154417]: 2023-07-20T02:42:23.334+0000 7f811a802700 -1 mds.l *** got signal Terminated *** 2023-07-20T02:42:25.097 INFO:journalctl@ceph.mds.l.smithi151.stdout:Jul 20 02:42:24 smithi151 bash[158676]: ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c-mds-l 2023-07-20T02:42:25.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9d6085e8-26a5-11ee-9b34-001a4aab830c@mds.l.service' 2023-07-20T02:42:25.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:42:25.141 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-20T02:42:25.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T02:42:25.461 INFO:teuthology.orchestra.run.smithi017.stdout:Deleting cluster with fsid: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:42:27.088 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T02:42:27.402 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:42:28.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T02:42:29.244 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:42:30.817 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:42:30.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:42:30.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:42:30.921 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-20T02:42:31.042 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi017/crash 2023-07-20T02:42:31.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T02:42:31.092 INFO:teuthology.orchestra.run.smithi017.stderr:tar: /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T02:42:31.092 INFO:teuthology.orchestra.run.smithi017.stderr:tar: Error is not recoverable: exiting now 2023-07-20T02:42:31.094 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi146/crash 2023-07-20T02:42:31.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T02:42:31.145 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T02:42:31.145 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now 2023-07-20T02:42:31.147 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi151/crash 2023-07-20T02:42:31.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T02:42:31.185 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T02:42:31.186 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-07-20T02:42:31.187 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-20T02:42:31.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-20T02:42:31.225 INFO:teuthology.orchestra.run.smithi017.stderr:grep: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph.log: No such file or directory 2023-07-20T02:42:31.227 INFO:tasks.cephadm:Compressing logs... 2023-07-20T02:42:31.227 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T02:42:31.270 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T02:42:31.272 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T02:42:31.298 INFO:teuthology.orchestra.run.smithi017.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T02:42:31.298 INFO:teuthology.orchestra.run.smithi017.stderr:: No such file or directory 2023-07-20T02:42:31.299 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.0.log 2023-07-20T02:42:31.300 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.g.log 2023-07-20T02:42:31.301 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.a.log 2023-07-20T02:42:31.301 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.6.log 2023-07-20T02:42:31.301 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.3.log 2023-07-20T02:42:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T02:42:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:: No such file or directory 2023-07-20T02:42:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.5.log 2023-07-20T02:42:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.2.log 2023-07-20T02:42:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.11.log 2023-07-20T02:42:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.z.log 2023-07-20T02:42:31.303 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.8.log 2023-07-20T02:42:31.303 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.11.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.z.log: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T02:42:31.303 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.f.log 2023-07-20T02:42:31.303 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.c.log 2023-07-20T02:42:31.305 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.6.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.3.log: 95.0% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.g.log.gz 2023-07-20T02:42:31.305 INFO:teuthology.orchestra.run.smithi017.stderr: 94.9% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.a.log.gz 2023-07-20T02:42:31.305 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.d.log 2023-07-20T02:42:31.305 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.x.log 2023-07-20T02:42:31.305 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.d.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.x.log: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T02:42:31.306 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.9.log 2023-07-20T02:42:31.306 INFO:teuthology.orchestra.run.smithi017.stderr: 94.9% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.d.log.gz 2023-07-20T02:42:31.306 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T02:42:31.306 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.j.log 2023-07-20T02:42:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T02:42:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:: No such file or directory 2023-07-20T02:42:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.10.log 2023-07-20T02:42:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.h.log 2023-07-20T02:42:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.e.log 2023-07-20T02:42:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.4.log 2023-07-20T02:42:31.308 INFO:teuthology.orchestra.run.smithi146.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T02:42:31.308 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.1.log 2023-07-20T02:42:31.308 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.k.log 2023-07-20T02:42:31.308 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T02:42:31.308 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.k.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log: 94.9% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.e.log.gz 2023-07-20T02:42:31.308 INFO:teuthology.orchestra.run.smithi146.stderr: 95.1% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.k.log.gz 2023-07-20T02:42:31.309 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.j.log: 94.9% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.j.log.gz 2023-07-20T02:42:31.309 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.a.log 2023-07-20T02:42:31.310 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.7.log 2023-07-20T02:42:31.310 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.b.log 2023-07-20T02:42:31.310 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.f.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.c.log: 91.3% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.z.log.gz 2023-07-20T02:42:31.311 INFO:teuthology.orchestra.run.smithi151.stderr: 94.2% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.f.log.gz 2023-07-20T02:42:31.311 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.i.log 2023-07-20T02:42:31.311 INFO:teuthology.orchestra.run.smithi151.stderr: 93.8% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.c.log.gz 2023-07-20T02:42:31.311 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.c.log 2023-07-20T02:42:31.312 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.7.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.b.log: 93.3% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.h.log.gz 2023-07-20T02:42:31.312 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.b.log 2023-07-20T02:42:31.313 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.b.log: 91.5% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T02:42:31.313 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.a.log: 91.5% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T02:42:31.313 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.y.log 2023-07-20T02:42:31.314 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.i.log: /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T02:42:31.318 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.y.log: 91.2% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.y.log.gz 2023-07-20T02:42:31.320 INFO:teuthology.orchestra.run.smithi146.stderr: 93.0% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.b.log.gz 2023-07-20T02:42:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log: 91.4% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T02:42:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.l.log 2023-07-20T02:42:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.l.log: 94.2% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.l.log.gz 2023-07-20T02:42:31.336 INFO:teuthology.orchestra.run.smithi151.stderr: 93.3% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mds.i.log.gz 2023-07-20T02:42:31.352 INFO:teuthology.orchestra.run.smithi151.stderr: 93.3% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.11.log.gz 2023-07-20T02:42:31.381 INFO:teuthology.orchestra.run.smithi146.stderr: 92.7% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.4.log.gz 2023-07-20T02:42:31.386 INFO:teuthology.orchestra.run.smithi017.stderr: 92.8% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.3.log.gz 2023-07-20T02:42:31.389 INFO:teuthology.orchestra.run.smithi146.stderr: 92.9% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.10.log.gz 2023-07-20T02:42:31.393 INFO:teuthology.orchestra.run.smithi017.stderr: 88.8% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mgr.x.log.gz 2023-07-20T02:42:31.394 INFO:teuthology.orchestra.run.smithi151.stderr: 92.9% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.8.log.gz 2023-07-20T02:42:31.394 INFO:teuthology.orchestra.run.smithi151.stderr: 92.9% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.2.log.gz 2023-07-20T02:42:31.401 INFO:teuthology.orchestra.run.smithi017.stderr: 92.8% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.9.log.gz 2023-07-20T02:42:31.410 INFO:teuthology.orchestra.run.smithi146.stderr: 92.1% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.b.log.gz 2023-07-20T02:42:31.414 INFO:teuthology.orchestra.run.smithi017.stderr: 92.7% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.6.log.gz 2023-07-20T02:42:31.415 INFO:teuthology.orchestra.run.smithi017.stderr: 93.1% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.0.log.gz 2023-07-20T02:42:31.416 INFO:teuthology.orchestra.run.smithi151.stderr: 92.7% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.5.log.gz 2023-07-20T02:42:31.435 INFO:teuthology.orchestra.run.smithi146.stderr: 92.8% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.7.log.gz 2023-07-20T02:42:31.436 INFO:teuthology.orchestra.run.smithi151.stderr: 92.1% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.c.log.gz 2023-07-20T02:42:31.437 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-07-20T02:42:31.437 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m0.149s 2023-07-20T02:42:31.437 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m0.519s 2023-07-20T02:42:31.437 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.045s 2023-07-20T02:42:31.458 INFO:teuthology.orchestra.run.smithi146.stderr: 93.3% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-osd.1.log.gz 2023-07-20T02:42:31.460 INFO:teuthology.orchestra.run.smithi146.stderr: 2023-07-20T02:42:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m0.171s 2023-07-20T02:42:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m0.564s 2023-07-20T02:42:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.048s 2023-07-20T02:42:31.642 INFO:teuthology.orchestra.run.smithi017.stderr: 91.1% -- replaced with /var/log/ceph/9d6085e8-26a5-11ee-9b34-001a4aab830c/ceph-mon.a.log.gz 2023-07-20T02:42:31.644 INFO:teuthology.orchestra.run.smithi017.stderr: 2023-07-20T02:42:31.644 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m0.358s 2023-07-20T02:42:31.644 INFO:teuthology.orchestra.run.smithi017.stderr:user 0m0.842s 2023-07-20T02:42:31.645 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m0.045s 2023-07-20T02:42:31.645 INFO:tasks.cephadm:Archiving logs... 2023-07-20T02:42:31.646 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi017/log 2023-07-20T02:42:31.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T02:42:31.827 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi146/log 2023-07-20T02:42:31.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T02:42:31.952 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi151/log 2023-07-20T02:42:31.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T02:42:32.062 INFO:tasks.cephadm:Removing cluster... 2023-07-20T02:42:32.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c --force 2023-07-20T02:42:32.353 INFO:teuthology.orchestra.run.smithi017.stdout:Deleting cluster with fsid: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:42:32.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c --force 2023-07-20T02:42:33.109 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:42:33.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9d6085e8-26a5-11ee-9b34-001a4aab830c --force 2023-07-20T02:42:33.856 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 9d6085e8-26a5-11ee-9b34-001a4aab830c 2023-07-20T02:42:34.311 INFO:tasks.cephadm:Removing cephadm ... 2023-07-20T02:42:34.312 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:34.334 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:34.353 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:34.371 INFO:tasks.cephadm:Teardown complete 2023-07-20T02:42:34.372 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi017 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T02:42:34.458 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=134cc45149f14948bba5dd2740569b10 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi017 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T02:42:34.463 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-20T02:42:34.471 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-20T02:42:34.494 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-20T02:42:34.495 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T02:42:34.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T02:42:34.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T02:42:34.617 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-20T02:42:34.625 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-20T02:42:34.626 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:42:34.629 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:42:34.631 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:42:34.646 INFO:teuthology.orchestra.run.smithi017.stderr:bash: ntpq: command not found 2023-07-20T02:42:34.646 INFO:teuthology.orchestra.run.smithi146.stderr:bash: ntpq: command not found 2023-07-20T02:42:34.647 INFO:teuthology.orchestra.run.smithi151.stderr:bash: ntpq: command not found 2023-07-20T02:42:34.650 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:42:34.651 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2023-07-20T02:42:34.651 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 -2143us[-2143us] +/- 110ms 2023-07-20T02:42:34.651 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 +1750us[+1750us] +/- 64ms 2023-07-20T02:42:34.651 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv03.front.sepia.ceph.com 3 6 377 64 -402us[ -224us] +/- 76ms 2023-07-20T02:42:34.652 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:42:34.653 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:42:34.654 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2023-07-20T02:42:34.654 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 -2158us[-2158us] +/- 110ms 2023-07-20T02:42:34.654 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 0 +1750us[+1750us] +/- 64ms 2023-07-20T02:42:34.654 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv03.front.sepia.ceph.com 3 6 377 2 -481us[ -325us] +/- 77ms 2023-07-20T02:42:34.655 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:42:34.655 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:42:34.655 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2023-07-20T02:42:34.655 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 -2127us[-2127us] +/- 110ms 2023-07-20T02:42:34.656 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 +1766us[+1766us] +/- 64ms 2023-07-20T02:42:34.656 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv03.front.sepia.ceph.com 3 6 377 4 -453us[ -250us] +/- 77ms 2023-07-20T02:42:34.656 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:42:34.656 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-20T02:42:34.665 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-20T02:42:34.665 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-20T02:42:34.673 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T02:42:34.796 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T02:42:34.893 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T02:42:34.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:42:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:42:35.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:42:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:42:35.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:42:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:42:35.088 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-20T02:42:35.100 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-20T02:42:35.110 INFO:teuthology.task.internal:Duration was 1842.905812 seconds 2023-07-20T02:42:35.112 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-20T02:42:35.141 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-20T02:42:35.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T02:42:35.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T02:42:35.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T02:42:35.192 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:42:35.195 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:42:35.197 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:42:35.629 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-20T02:42:35.629 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2023-07-20T02:42:35.630 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T02:42:35.657 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2023-07-20T02:42:35.657 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T02:42:35.678 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2023-07-20T02:42:35.679 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T02:42:35.702 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-20T02:42:35.702 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T02:42:35.705 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T02:42:35.721 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T02:42:35.788 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-20T02:42:35.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T02:42:35.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T02:42:35.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T02:42:35.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-20T02:42:35.899 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-20T02:42:35.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T02:42:35.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T02:42:35.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T02:42:35.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-20T02:42:35.992 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:42:35.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:42:35.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:42:36.021 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2023-07-20T02:42:36.021 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2023-07-20T02:42:36.048 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2023-07-20T02:42:36.080 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:42:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:42:36.098 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:42:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:42:36.115 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:42:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:42:36.137 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-20T02:42:36.157 INFO:teuthology.task.internal:Transferring archived files... 2023-07-20T02:42:36.158 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi017 2023-07-20T02:42:36.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T02:42:36.211 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi146 2023-07-20T02:42:36.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T02:42:36.248 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291/remote/smithi151 2023-07-20T02:42:36.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T02:42:36.286 INFO:teuthology.task.internal:Removing archive directory... 2023-07-20T02:42:36.287 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T02:42:36.290 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T02:42:36.292 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T02:42:36.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-20T02:42:36.349 INFO:teuthology.task.internal:Not uploading archives. 2023-07-20T02:42:36.350 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-20T02:42:36.372 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-20T02:42:36.373 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T02:42:36.375 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T02:42:36.378 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T02:42:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 02:42 /home/ubuntu/cephtest 2023-07-20T02:42:36.392 INFO:teuthology.orchestra.run.smithi146.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 02:42 /home/ubuntu/cephtest 2023-07-20T02:42:36.395 INFO:teuthology.orchestra.run.smithi151.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 02:42 /home/ubuntu/cephtest 2023-07-20T02:42:36.396 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-20T02:42:36.407 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-20T02:42:36.433 INFO:teuthology.nuke:Checking targets against current locks 2023-07-20T02:42:36.464 DEBUG:teuthology.nuke:shortname: smithi017 2023-07-20T02:42:36.465 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:42:36.491 DEBUG:teuthology.nuke:shortname: smithi146 2023-07-20T02:42:36.491 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:42:36.495 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:49.586796', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T02:42:36.516 DEBUG:teuthology.nuke:shortname: smithi151 2023-07-20T02:42:36.517 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:42:36.523 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:49.584227', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T02:42:36.544 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:49.585686', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T02:42:36.556 INFO:teuthology.orchestra.console:Power off smithi017 2023-07-20T02:42:36.557 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T02:42:36.580 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:36.597 INFO:teuthology.orchestra.console:Power off smithi146 2023-07-20T02:42:36.599 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T02:42:36.617 INFO:teuthology.orchestra.console:Power off smithi151 2023-07-20T02:42:36.618 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T02:42:36.628 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:36.642 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:40.628 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:40.654 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:40.666 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:44.743 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:44.769 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:44.779 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:44.957 INFO:teuthology.orchestra.console:Power off for smithi017 completed 2023-07-20T02:42:44.982 INFO:teuthology.orchestra.console:Power off for smithi146 completed 2023-07-20T02:42:48.896 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:42:49.107 INFO:teuthology.orchestra.console:Power off for smithi151 completed 2023-07-20T02:42:49.213 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} duration: 1842.9058120250702 failure_reason: 'Command failed on smithi017 with status 2: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=134cc45149f14948bba5dd2740569b10 status: fail success: false 2023-07-20T02:42:49.214 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T02:42:49.307 INFO:teuthology.run:FAIL